chore: prepare repository for Predix public release

- Rebrand from RD-Agent to Predix for EUR/USD quantitative trading
- Update all documentation to English
- Remove Microsoft-specific references
- Clean up temporary files and backups
- Update LICENSE, README, and configuration for PredixAI organization

Breaking changes:
- Project name changed from 'rdagent' to 'predix' in pyproject.toml
- All Microsoft and RD-Agent branding replaced with Predix
- Documentation completely rewritten for EUR/USD focus

Documentation:
- README.md: Professional English documentation with installation, quick start, CLI reference
- CHANGELOG.md: Cleaned up, references upstream RD-Agent for historical changes
- CODE_OF_CONDUCT.md: Switched to Contributor Covenant v2.0
- SECURITY.md: Predix-specific vulnerability reporting process
- SUPPORT.md: Updated support channels (nico@predix.io, GitHub Discussions)
- CONTRIBUTING.md: Adapted for Predix project
- docs/: Sphinx configuration updated for Predix branding

Configuration:
- pyproject.toml: Updated project metadata, keywords, URLs for PredixAI
- .gitignore: Comprehensive Python/gitignore template
- Makefile: Updated CI pages URL
- setup_predix_eurusd.sh: Translated to English

Cleanup:
- Deleted log files, caches, __pycache__ directories
- Removed backup files (*.backup_*)
- Cleaned web/node_modules

.bumpversion.cfg

@@ -1,6 +0,0 @@
-[bumpversion]
-current_version = 0.0.0
-commit = True
-tag = True
-
-[bumpversion:file:pyproject.toml]

.commitlintrc.js

@@ -1,21 +0,0 @@
-module.exports = {
-    extends: ["@commitlint/config-conventional"],
-    rules: {
-        // Configuration Format: [level, applicability, value]
-        // level: Error level, usually expressed as a number:
-        //     0 - disable rule
-        //     1 - Warning (does not prevent commits)
-        //     2 - Error (will block the commit)
-        // applicability: the conditions under which the rule applies, commonly used values:
-        //     “always” - always apply the rule
-        //     “never” - never apply the rule
-        // value: the specific value of the rule, e.g. a maximum length of 100.
-        // Refs: https://commitlint.js.org/reference/rules-configuration.html
-      "header-max-length": [2, "always", 100],
-      "type-enum": [
-        2,
-        "always",
-        ["build", "chore", "ci", "docs", "feat", "fix", "perf", "refactor", "revert", "style", "test", "Release-As"]
-      ]
-    }
-  };

.env.example

@@ -1,59 +0,0 @@
-"""
-This file is a template for the .env file.
-
-Please copy this file to .env and fill in the values.
-
-For more information about configuration options, please refer to the documentation
-
-"""
-
-# ==========================================
-# Global configs:
-MAX_RETRY=10
-RETRY_WAIT_SECONDS=20
-# ==========================================
-
-
-# ==========================================
-# Backend Configuration
-# ==========================================
-# BACKEND=rdagent.oai.backend.LiteLLMAPIBackend
-# ==========================================
-
-# ==========================================
-# Backend Configuration (choose one)
-# ==========================================
-
-# 1. Set universal API key
-# CHAT_MODEL="gpt-4o"
-# EMBEDDING_MODEL="text-embedding-3-small"
-# OPENAI_API_BASE="https://your-endpoint.com/v1"
-# OPENAI_API_KEY="sk-your-api-key-here"
-
-# 2. Set separate API KEY
-# Chat configuration
-OPENAI_API_KEY="sk-chat-key"
-OPENAI_API_BASE="https://xxx-litellm.com/v1"
-CHAT_MODEL='gpt-4o'
-
-# Embedding configuration (using other service)
-# Use siliconflow as example, pay attention to the litellm_proxy prefix
-LITELLM_PROXY_API_KEY="sk-embedding-service-key"
-LITELLM_PROXY_API_BASE="https://api.siliconflow.cn/v1"
-EMBEDDING_MODEL="litellm_proxy/BAAI/bge-large-en-v1.5"
-# ==========================================
-
-# ==========================================
-# Other Configuration 
-# ==========================================
-# CHAT_AZURE_API_BASE=<for_Azure_user>
-# CHAT_AZURE_API_VERSION=<for_Azure_user>
-
-# EMBEDDING_AZURE_API_BASE=<for_Azure_user>
-# EMBEDDING_AZURE_API_VERSION=<for_Azure_user>
-
-# Cache Setting (Optional):
-# USE_CHAT_CACHE=True
-# USE_EMBEDDING_CACHE=True
-# Senario Configs:
-# ==========================================

.github/FUNDING.yml

@@ -1,2 +0,0 @@
-github:
-  - MIIC-finance

.github/ISSUE_TEMPLATE/bug-report.md

@@ -1,51 +0,0 @@
----
-name: "\U0001F41B Bug Report"
-about: Submit a bug report to help us improve RD-Agent
-labels: bug
-
----
-
-## 🐛 Bug Description
-
-<!-- A clear and concise description of what the bug is. -->
-
-## To Reproduce
-
-Steps to reproduce the behavior:
-
-1.
-2.
-3.
-
-
-## Expected Behavior
-
-<!-- A clear and concise description of what you expected to happen. -->
-
-## Screenshot
-
-<!-- A screenshot of the error message or anything shouldn't appear-->
-
-## Environment
-
-**Note**: Users can run `rdagent collect_info` to get system information and paste it directly here.
-
- - Name of current operating system:
- - Processor architecture:
- - System, version, and hardware information:
- - Version number of the system:
- - Python version:
- - Container ID:
- - Container Name:
- - Container Status:
- - Image ID used by the container:
- - Image tag used by the container:
- - Container port mapping:
- - Container Label:
- - Startup Commands:
- - RD-Agent version:
- - Package version:
-
-## Additional Notes
-
-<!-- Add any other information about the problem here. -->

.github/ISSUE_TEMPLATE/documentation.md

@@ -1,9 +0,0 @@
----
-name: "\U0001F4D6 Documentation"
-about: Report an issue related to documentation
-
----
-
-## 📖 Documentation
-
-<!-- Please specify whether it's tutorial part or API reference part, and describe it.-->

.github/ISSUE_TEMPLATE/feature-request.md

@@ -1,25 +0,0 @@
----
-name: "\U0001F31FFeature Request"
-about: Request for a new RD-Agent feature
-labels: enhancement
-
----
-
-## 🌟 Feature Description
-<!-- A clear and concise description of the feature proposal -->
-
-## Motivation
-
-1. Application scenario
-2. Related works (Papers, Github repos etc.):
-3. Any other relevant and important information:
-
-<!-- Please describe why the feature is important. -->
-
-## Alternatives
-
-<!-- A short description of any alternative solutions or features you've considered. -->
-
-## Additional Notes
-
-<!-- Add any other context or screenshots about the feature request here. -->

.github/ISSUE_TEMPLATE/question.md

@@ -1,10 +0,0 @@
----
-name: "❓Questions & Help"
-about: Have some questions? We can offer help.
-labels: question
-
----
-
-## ❓ Questions and Help
-
-We sincerely suggest you to carefully read the [documentation](http://rdagent.readthedocs.io/). After that, if you still feel puzzled, please describe the question clearly under this issue.

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,34 +0,0 @@
-<!--- Thank you for submitting a Pull Request! In order to make our work smoother. -->
-<!--- please make sure your Pull Request meets the following requirements: -->
-<!---   1. Provide a general summary of your changes in the Title above; -->
-<!---   2. Add appropriate prefixes to titles, such as `build:`, `chore:`, `ci:`, `docs:`, `feat:`, `fix:`, `perf:`, `refactor:`, `revert:`, `style:`, `test:`(Ref: https://www.conventionalcommits.org/). -->
-<!--- Category: -->
-<!--- Patch Updates: `fix:` -->
-<!---   Example: fix(auth): correct login validation issue -->
-<!--- minor update (introduces new functionality): `feat` -->
-<!---   Example: feature(parser): add ability to parse arrays -->
-<!--- major update(destructive update): Include BREAKING CHANGE in the commit message footer, or add `! ` in the commit footer to indicate that there is a destructive update. -->
-<!---   Example: feat(auth)! : remove support for old authentication method -->
-<!--- Other updates: `build:`, `chore:`, `ci:`, `docs:`, `perf:`, `refactor:`, `revert:`, `style:`, `test:`. -->
-
-## Description
-<!--- Describe your changes in detail -->
-
-## Motivation and Context
-<!--- Are there any related issues? If so, please put the link here. -->
-<!--- Why is this change required? What problem does it solve? -->
-
-## How Has This Been Tested?
-<!---  Put an `x` in all the boxes that apply: --->
-- [ ] If you are adding a new feature, test on your own test scripts.
-
-<!--- **ATTENTION**: If you are adding a new feature, please make sure your codes are **correctly tested**. If our test scripts do not cover your cases, please provide your own test scripts under the `tests` folder and test them. More information about test scripts can be found [here](https://docs.python.org/3/library/unittest.html#basic-example), or you could refer to those we provide under the `tests` folder. -->
-
-## Screenshots of Test Results (if appropriate):
-1. Your own tests:
-
-## Types of changes
-<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
-- [ ] Fix bugs
-- [ ] Add new feature
-- [ ] Update documentation

.github/dependabot.yml

@@ -1,19 +0,0 @@
-updates:
-  - commit-message:
-      prefix: build(actions)
-    directory: /
-    package-ecosystem: github-actions
-    schedule:
-      interval: weekly
-  - commit-message:
-      prefix: build(requirements)
-    directory: /
-    groups:
-      dev:
-        dependency-type: development
-      prod:
-        dependency-type: production
-    package-ecosystem: pip
-    schedule:
-      interval: weekly
-version: 2

.github/workflows/ci.yml

@@ -1,70 +0,0 @@
-concurrency:
-  cancel-in-progress: true
-  group: ${{ github.workflow }}-${{ github.ref }}
-jobs:
-  ci:
-    if: ${{ !cancelled() && ! failure() }}
-    needs: dependabot
-    runs-on: ubuntu-latest
-    steps:
-      - name: checkout
-        uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-          submodules: recursive
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v5
-        with:
-          cache: pip
-          python-version: ${{ matrix.python-version }}
-      - run: env | sort
-      - run: make dev
-      - name: lint test docs and build
-        run: make lint docs-gen test-offline # test docs build
-    strategy:
-      matrix:
-        python-version:
-          - '3.10'
-          - '3.11'
-  dependabot:
-    if: ${{ github.actor == 'dependabot[bot]' && startsWith(github.head_ref, 'dependabot/pip/') }}
-    permissions:
-      contents: write
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-          ref: ${{ github.head_ref }}
-      - name: Set up Git
-        run: |
-          git config --global user.name github-actions
-          git config --global user.email github-actions@github.com
-      - name: Set up Python with multiple versions.
-        uses: actions/setup-python@v5
-        with:
-          cache: pip
-          python-version: |
-            3.10
-            3.11
-      - name: Install pipenv using pipx
-        run: pipx install pipenv
-      - name: Generate constraints for all supported Python versions
-        run: |
-          CI= PYTHON_VERSION=3.10 make constraints
-          CI= PYTHON_VERSION=3.11 make constraints
-      - name: Push changes if applicable
-        run: |
-          if [[ -n `git status --porcelain` ]]; then
-            git commit -a -m "build: Update constraints for dependabot."
-            git push
-          fi
-name: CI
-on:
-  pull_request:
-    types:
-      - opened
-      - synchronize
-  push:
-    branches:
-      - main

.github/workflows/pr.yml

@@ -1,35 +0,0 @@
-name: Lint pull request title
-
-on:
-  pull_request:
-    types:
-      - opened
-      - synchronize
-      - reopened
-      - edited
-
-concurrency:
-  cancel-in-progress: true
-  group: ${{ github.workflow }}-${{ github.ref }}
-
-jobs:
-  lint-title:
-    runs-on: ubuntu-latest
-    steps:
-      # This step is necessary because the lint title uses the .commitlintrc.js file in the project root directory.
-      - name: Checkout Repository
-        uses: actions/checkout@v4
-
-      - name: Setup Node.js
-        uses: actions/setup-node@v4
-        with:
-          node-version: '16'
-
-      - name: Install commitlint
-        run: npm install --save-dev @commitlint/{config-conventional,cli}
-
-      - name: Validate PR Title with commitlint
-        env:
-          BODY: ${{ github.event.pull_request.title }}
-        run: |
-          echo "$BODY" | npx commitlint --config .commitlintrc.js

.github/workflows/readthedocs-preview.yml

@@ -1,17 +0,0 @@
-concurrency:
-  cancel-in-progress: true
-  group: ${{ github.workflow }}-${{ github.ref }}
-jobs:
-  documentation-links:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: readthedocs/actions/preview@v1
-        with:
-          project-slug: RDAgent
-name: Read the Docs Pull Request Preview
-on:
-  pull_request_target:
-    types:
-      - opened
-permissions:
-  pull-requests: write

.github/workflows/release.yml

@@ -1,50 +0,0 @@
-name: Release
-on:
-  push:
-    branches:
-      - main
-permissions:
-  contents: read
-jobs:
-  release_and_publish:
-    permissions:
-      contents: write
-      pull-requests: read
-    runs-on: ubuntu-latest
-    steps:
-      - name: Release please
-        id: release_please
-        uses: googleapis/release-please-action@v4
-        with:
-          # The current PAT (personal access token) was created on 2024-08-05,
-          # since the maximum validity of PAT is 1 year, you need to change the PAT before 2025-08-05.
-          token: ${{ secrets.PAT }}
-          release-type: simple
-      - uses: actions/checkout@v4
-        if: ${{ steps.release_please.outputs.release_created }}
-        with:
-          fetch-depth: 0
-      - name: Set up Python
-        if: ${{ steps.release_please.outputs.release_created }}
-        uses: actions/setup-python@v5
-        with:
-          cache: pip
-          python-version: '3.10'
-      - name: Install dependencies
-        if: ${{ steps.release_please.outputs.release_created }}
-        run: |
-          python -m pip install --upgrade pip
-          pip install setuptools wheel twine  #  better-exceptions(optional for debug)
-      - run: env | sort
-        if: ${{ steps.release_please.outputs.release_created }}
-      - run: make dev
-        if: ${{ steps.release_please.outputs.release_created }}
-      - run: make build
-        if: ${{ steps.release_please.outputs.release_created }}
-      - name: upload
-        if: ${{ steps.release_please.outputs.release_created }}
-        env:
-          TWINE_USERNAME: __token__
-          TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
-        run: |
-          make upload

.gitignore

@@ -1,20 +1,13 @@
-# Custom
-*.swp
-.DS_Store
-Pipfile
-public
-release-notes.md
-typescript*
+# Environment
+.env
+.env.*
+!.env.example
 
-# Byte-compiled / optimized / DLL files
+# Python
 __pycache__/
 *.py[cod]
 *$py.class
-
-# C extensions
 *.so
-
-# Distribution / packaging
 .Python
 build/
 develop-eggs/
@@ -28,153 +21,47 @@ parts/
 sdist/
 var/
 wheels/
-pip-wheel-metadata/
-share/python-wheels/
 *.egg-info/
 .installed.cfg
 *.egg
-MANIFEST
 
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
+# Virtual environments
+venv/
+ENV/
+env/
+.venv/
 
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
 
-# Unit test / coverage reports
+# Testing
+.pytest_cache/
+.coverage
 htmlcov/
 .tox/
 .nox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
-.hypothesis/
-.pytest_cache/
 
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
+# Logs
 *.log
-/log*/
-local_settings.py
-db.sqlite3
-db.sqlite3-journal
+log/
 
-# Flask stuff:
-instance/
-.webassets-cache
+# Cache
+pickle_cache/
+prompt_cache.db
+.cache/
 
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-target/
-
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-.python-version
-
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow
-__pypackages__/
-
-# Celery stuff
-celerybeat-schedule
-celerybeat.pid
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
-.env*
-*.env
-.venv
-^env/
-venv/
-ENV/
-env.bak/
-venv.bak/
-
-# Spyder project settings
-.spyderproject
-.spyproject
-
-# Rope project settings
-.ropeproject
-
-# mkdocs documentation
-/site
-
-# mypy
-.mypy_cache/
-.dmypy.json
-dmypy.json
-
-# Pyre type checker
-.pyre/
-
-# all pkl files
-*.pkl
-
-# all h5 files
-*.h5
-
-# all vs-code files
-.vscode/
-
-# reports
-reports/
-
-# git_ignore_folder
+# Generated/processed data
 git_ignore_folder/
+data_raw/
 
-#cache
-*cache*/
-*cache.json
+# Build artifacts
+*.manifest
+*.spec
 
-# DB files
-*.db
-
-# Docker
-factor_template/mlruns/
-env_tpl
-mlruns/
-
-# possible output from coder or runner
-*.pth
-*qlib_res.csv
-
-# shell script
-*.out
-/*.sh
-.aider*
-rdagent/app/benchmark/factor/example.json
-
-# UI Server resources
-videos/
-static/
+# Local scripts (generated)
+convert_1min.py
+import_1min_qlib.py

.readthedocs.yaml

@@ -1,38 +0,0 @@
-# .readthedocs.yml
-# Read the Docs configuration file
-# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
-
-# Required
-version: 2
-
-# Set the version of Python and other tools you might need
-build:
-  os: ubuntu-22.04
-  tools:
-    python: "3.10"
-  # During the build process, you need to fetch tags, and since the default command to read the docs only pulls shallow code, it will cause an error.
-  # So we added the `git fetch --tags --unshallow || true` command to fetch the full tag record.
-  # Adding this command overrides the default command, so we copied it over to make sure the build was successful.
-  commands:
-    - python -mvirtualenv $READTHEDOCS_VIRTUALENV_PATH 
-    - python -m pip install --upgrade --no-cache-dir pip setuptools 
-    - python -m pip install --upgrade --no-cache-dir sphinx 
-    - python -m pip install --exists-action=w --no-cache-dir -r requirements/docs.txt 
-    - python -m pip install --upgrade --upgrade-strategy only-if-needed --no-cache-dir . 
-    - git fetch --tags --unshallow || true
-    - mkdir -p $READTHEDOCS_OUTPUT/html/
-    - python -m sphinx -T -b html -d _build/doctrees -D language=en ./docs $READTHEDOCS_OUTPUT/html
-
-# Build documentation in the docs/ directory with Sphinx
-sphinx:
-  configuration: docs/conf.py
-
-# Build all formats
-formats: all
-
-# Optionally set the version of Python and requirements required to build your docs
-python:
-  install:
-    - requirements: requirements/docs.txt
-    - method: pip
-      path: .

.streamlit/config.toml

@@ -1,2 +0,0 @@
-[client]
-showSidebarNavigation = false

CHANGELOG.md

@@ -1,446 +1,23 @@
 # Changelog
 
-## [0.6.1](https://github.com/microsoft/RD-Agent/compare/v0.6.0...v0.6.1) (2025-06-28)
+All notable changes to Predix will be documented in this file.
 
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-### Bug Fixes
+## [Unreleased]
 
-* fix mount ([#1001](https://github.com/microsoft/RD-Agent/issues/1001)) ([4ae2f13](https://github.com/microsoft/RD-Agent/commit/4ae2f1303dfcbaea53d459be7c8e85bf85ce5f4f))
-* handle the bug of wrong dag_parant index ([#996](https://github.com/microsoft/RD-Agent/issues/996)) ([bda12ff](https://github.com/microsoft/RD-Agent/commit/bda12ffecf9ae116e0d04eece0c6a1b61413d916))
-* improve log folder sorting and selection UX ([#993](https://github.com/microsoft/RD-Agent/issues/993)) ([b116807](https://github.com/microsoft/RD-Agent/commit/b11680777f116b6c40f9e535e0da10c186c95050))
+### Changed
+- Rebranded from RD-Agent to Predix for EUR/USD quantitative trading
+- Updated project metadata for PredixAI organization
 
-## [0.6.0](https://github.com/microsoft/RD-Agent/compare/v0.5.0...v0.6.0) (2025-06-26)
+### Fixed
+- Removed all Chinese stock references, replaced with EUR/USD 1min FX data
+- Migrated to 1min EURUSD data (2020-2026)
+- Injected MultiIndex warning into factor interface prompt
 
+---
 
-### Features
+## Historical Changes (from RD-Agent upstream)
 
-* async mechanism for multi-trace ([#981](https://github.com/microsoft/RD-Agent/issues/981)) ([9e60c32](https://github.com/microsoft/RD-Agent/commit/9e60c32cf348481eb55617809c059c359d7603b8))
-
-
-### Bug Fixes
-
-* add async to direct_exp_gen avoid infinite loop ([#992](https://github.com/microsoft/RD-Agent/issues/992)) ([78c203d](https://github.com/microsoft/RD-Agent/commit/78c203d8eefbba67fc120b35cb25e85b2200ac49))
-* docker container cleanup to prevent accumulation and system slowdown ([#975](https://github.com/microsoft/RD-Agent/issues/975)) ([05cf094](https://github.com/microsoft/RD-Agent/commit/05cf094913e48c903c8a4476d6c609d8bfa10681))
-* fix a bug and update the docs ([#978](https://github.com/microsoft/RD-Agent/issues/978)) ([d1ae9e1](https://github.com/microsoft/RD-Agent/commit/d1ae9e1dcc2ccd1ffe05cb1c6db3e905fa70425c))
-* merge datascience v3 and v2 ([#974](https://github.com/microsoft/RD-Agent/issues/974)) ([1ba7548](https://github.com/microsoft/RD-Agent/commit/1ba754853ce2010ce1cb0bbd217b67689fa1ebdf))
-* refine details ([#979](https://github.com/microsoft/RD-Agent/issues/979)) ([25caa3d](https://github.com/microsoft/RD-Agent/commit/25caa3d00c255286dce27915b9355987b87ed2e8))
-* refine prompt ([#987](https://github.com/microsoft/RD-Agent/issues/987)) ([76df96e](https://github.com/microsoft/RD-Agent/commit/76df96ee88212a8aee7f518b9cacf80591dc2939))
-
-## [0.5.0](https://github.com/microsoft/RD-Agent/compare/v0.4.0...v0.5.0) (2025-06-18)
-
-
-### Features
-
-* add a check for whether values in score_df are NaN ([#756](https://github.com/microsoft/RD-Agent/issues/756)) ([d9cc780](https://github.com/microsoft/RD-Agent/commit/d9cc78098beb27f3a1bf2f2d461302db177b7d41))
-* add competition level filter and extract constants to utils ([#869](https://github.com/microsoft/RD-Agent/issues/869)) ([b40b605](https://github.com/microsoft/RD-Agent/commit/b40b6055368e6c72d8435352104b1c281b06da7f))
-* add DocDev for auto-generating workspace documentation ([#781](https://github.com/microsoft/RD-Agent/issues/781)) ([bcba6ea](https://github.com/microsoft/RD-Agent/commit/bcba6eac32684ebb267c93b4e85dbfa9561d15d1))
-* add drafting pipeline ([#832](https://github.com/microsoft/RD-Agent/issues/832)) ([efedddf](https://github.com/microsoft/RD-Agent/commit/efedddf39bc19221fdffc2e39ee0a09097fc82b0))
-* add last_exp_fb to DSTrace and update feedback retrieval usage ([#910](https://github.com/microsoft/RD-Agent/issues/910)) ([10531fd](https://github.com/microsoft/RD-Agent/commit/10531fda9438c6915b26d5013bd2413e1333ceb9))
-* add mlflow logger in RD loop to log ([#815](https://github.com/microsoft/RD-Agent/issues/815)) ([b91b54f](https://github.com/microsoft/RD-Agent/commit/b91b54f355c26b751087d0c14774f466e82866de))
-* add naive experiment generator and update proposal configurations ([#759](https://github.com/microsoft/RD-Agent/issues/759)) ([75494f4](https://github.com/microsoft/RD-Agent/commit/75494f4fed5bc845acfd7f7bacef385f0f96c514))
-* add RD-Agent-Quant scenario ([#838](https://github.com/microsoft/RD-Agent/issues/838)) ([6e42d52](https://github.com/microsoft/RD-Agent/commit/6e42d523a85df67aa13927abbf0894564c71880e))
-* add reasoning_effort parameter to LiteLLMAPIBackend and LLMSett… ([#754](https://github.com/microsoft/RD-Agent/issues/754)) ([113889f](https://github.com/microsoft/RD-Agent/commit/113889fefe9b09aaea1b564704c81664b8f77ec5))
-* add reviewer in feedback ([#765](https://github.com/microsoft/RD-Agent/issues/765)) ([1a95bee](https://github.com/microsoft/RD-Agent/commit/1a95bee6aa6bc6f45fdeb484f3a6f81caa273038))
-* advanced checkpoint selectors ([#790](https://github.com/microsoft/RD-Agent/issues/790)) ([50ea033](https://github.com/microsoft/RD-Agent/commit/50ea0336e93d8cb39fb871e81a3f61abdf293bc7))
-* archive python and csv files in workspace to maintain results ([#814](https://github.com/microsoft/RD-Agent/issues/814)) ([67d0e01](https://github.com/microsoft/RD-Agent/commit/67d0e01e7c9237da1371d93cbf9d86f5f46faac4))
-* checkpoint selection ([#744](https://github.com/microsoft/RD-Agent/issues/744)) ([a15a06a](https://github.com/microsoft/RD-Agent/commit/a15a06ad643977db59d7cac9da52e637cf80395a))
-* custom data ([#810](https://github.com/microsoft/RD-Agent/issues/810)) ([6322916](https://github.com/microsoft/RD-Agent/commit/632291608cf605bd8bcfcab0017824823bdecdb8))
-* dump model ([#776](https://github.com/microsoft/RD-Agent/issues/776)) ([b49481e](https://github.com/microsoft/RD-Agent/commit/b49481e073e6f536d2b1b3bd2d01229ed05abdea))
-* enable to set different version of idea-proposal for multi traces ([#895](https://github.com/microsoft/RD-Agent/issues/895)) ([236c28f](https://github.com/microsoft/RD-Agent/commit/236c28f29c6bc5da62129632e464bbc32056ebdb))
-* enhance compatibility with more LLM models ([#905](https://github.com/microsoft/RD-Agent/issues/905)) ([8800624](https://github.com/microsoft/RD-Agent/commit/8800624ad4749d6e798785a082c9f94c306792ef))
-* idea pool integrated to exp_gen & add timer to RD-Agent & pause-resume to RD-loops ([#795](https://github.com/microsoft/RD-Agent/issues/795)) ([e62aefa](https://github.com/microsoft/RD-Agent/commit/e62aefa56e34ff45a8ed033f7bf28b95c8e63656))
-* joblib cache ([#749](https://github.com/microsoft/RD-Agent/issues/749)) ([83a0411](https://github.com/microsoft/RD-Agent/commit/83a041148ff908871b1906f9e6889d80ab513412))
-* log api status to mlflow ([#860](https://github.com/microsoft/RD-Agent/issues/860)) ([049921b](https://github.com/microsoft/RD-Agent/commit/049921beb0b4ed0ba1ab7508d9857d2c1e729349))
-* log reaching max time limit before breaking CoSTEER evolution ([#921](https://github.com/microsoft/RD-Agent/issues/921)) ([837fff2](https://github.com/microsoft/RD-Agent/commit/837fff29096fefe1369d386ef8a860395b737173))
-* merge failed and successful traces together ([#766](https://github.com/microsoft/RD-Agent/issues/766)) ([3a2aa8c](https://github.com/microsoft/RD-Agent/commit/3a2aa8cf0102647950b2dfc0007c118b0c799cd4))
-* merge selectively ([#888](https://github.com/microsoft/RD-Agent/issues/888)) ([06ba314](https://github.com/microsoft/RD-Agent/commit/06ba314ff0f91e7e78e8d456c719ac3194a8c774))
-* multi-trace online merge ([#886](https://github.com/microsoft/RD-Agent/issues/886)) ([2112d67](https://github.com/microsoft/RD-Agent/commit/2112d676d0938de6fea163b2e5eb9c36771e7041))
-* new proposal (structured outputs) prompts ([#887](https://github.com/microsoft/RD-Agent/issues/887)) ([150796a](https://github.com/microsoft/RD-Agent/commit/150796aaa72eaa5037fd7db8e785058fbc4d4967))
-* parallel loop running based on asyncio ([#932](https://github.com/microsoft/RD-Agent/issues/932)) ([c63e207](https://github.com/microsoft/RD-Agent/commit/c63e2071f3179feef69f88061c0172cb5c3157f2))
-* propose hypothesis across multiple parts in pipeline ([#827](https://github.com/microsoft/RD-Agent/issues/827)) ([acb0e21](https://github.com/microsoft/RD-Agent/commit/acb0e21a331410d044849e12e2887f41e5ff1c3a))
-* pull image with progress ([#777](https://github.com/microsoft/RD-Agent/issues/777)) ([5cad086](https://github.com/microsoft/RD-Agent/commit/5cad0860204ede974533dc7bdc9808cfd135fa24))
-* raise error when timeout in api call ([#793](https://github.com/microsoft/RD-Agent/issues/793)) ([eafd4df](https://github.com/microsoft/RD-Agent/commit/eafd4dfc6263f19a8cdaf27498a1d07b43815306))
-* raise policy violation ([#894](https://github.com/microsoft/RD-Agent/issues/894)) ([5b9d007](https://github.com/microsoft/RD-Agent/commit/5b9d0072aebe15369e9a0010af83e71684baeae7))
-* reanalyze competition info & pipeline coding evaluator prompt ([#837](https://github.com/microsoft/RD-Agent/issues/837)) ([f7b5258](https://github.com/microsoft/RD-Agent/commit/f7b52580080c75d311355bcc6193b49495801809))
-* refine merge ([#842](https://github.com/microsoft/RD-Agent/issues/842)) ([99463b4](https://github.com/microsoft/RD-Agent/commit/99463b46819b3a0dcb2bb12a823a9cdf7ec560b4))
-* refine prompt ([#760](https://github.com/microsoft/RD-Agent/issues/760)) ([a91b182](https://github.com/microsoft/RD-Agent/commit/a91b182c4c9510eb34e4aab956588e909fa5d70b))
-* replace hard-coded cache paths with dynamic cache_path config ([#952](https://github.com/microsoft/RD-Agent/issues/952)) ([db56894](https://github.com/microsoft/RD-Agent/commit/db568947f1084a80d603718f5a13fdbd72b90a47))
-* revert draft stage into a soft decay in hypothesis selection ([#849](https://github.com/microsoft/RD-Agent/issues/849)) ([d41db0c](https://github.com/microsoft/RD-Agent/commit/d41db0ca357b07091825ebd9d18c303b6db3cc6a))
-* trace merging ([#836](https://github.com/microsoft/RD-Agent/issues/836)) ([a3d5473](https://github.com/microsoft/RD-Agent/commit/a3d547369e408a05cff570c1239b6320be40418d))
-* truncate by time ([#863](https://github.com/microsoft/RD-Agent/issues/863)) ([2b9427a](https://github.com/microsoft/RD-Agent/commit/2b9427ae036ffe1e28a717502f45500fe91fe5ac))
-* update prompt to improve json respond format of some LLM models ([#928](https://github.com/microsoft/RD-Agent/issues/928)) ([0b84709](https://github.com/microsoft/RD-Agent/commit/0b84709e59c7abb9754961cd17cc9673fcf508aa))
-* using different chat model in different part ([#822](https://github.com/microsoft/RD-Agent/issues/822)) ([c052ea6](https://github.com/microsoft/RD-Agent/commit/c052ea6d1f8948183a4a6ebc873ec01b57373cce))
-
-
-### Bug Fixes
-
-* 'DSProposalV2ExpGen' object has no attribute 'COMPONENT_TASK_MAP… ([#950](https://github.com/microsoft/RD-Agent/issues/950)) ([e353895](https://github.com/microsoft/RD-Agent/commit/e353895251f231fee85abdcb1b22b022a577af77))
-* adapting UI to mock trace ([#841](https://github.com/microsoft/RD-Agent/issues/841)) ([8a5754c](https://github.com/microsoft/RD-Agent/commit/8a5754c9b9c9410d0943aeed777a93c13422e54a))
-* add missing semicolon after chmod in env shell command ([#955](https://github.com/microsoft/RD-Agent/issues/955)) ([1128eaa](https://github.com/microsoft/RD-Agent/commit/1128eaa89ec1dcab4a05ef50d64c7f7e6aae88a8))
-* add time to timer when api timeout bug ([#826](https://github.com/microsoft/RD-Agent/issues/826)) ([f45d6ae](https://github.com/microsoft/RD-Agent/commit/f45d6ae6595c1c39b389485b637a0ae53ffc8782))
-* add wait_retry to exp_gen v2 ([#783](https://github.com/microsoft/RD-Agent/issues/783)) ([b9fb7cf](https://github.com/microsoft/RD-Agent/commit/b9fb7cf4e3070062d91b5b67d0f10d6266b45142))
-* adjust ds_trace lookup and add stderr redirect to mlebench command ([#853](https://github.com/microsoft/RD-Agent/issues/853)) ([4e53108](https://github.com/microsoft/RD-Agent/commit/4e53108e020db719b39cba3a67e0c6dae3de19cf))
-* align competion_full_desc and scenario_all_desc, remove redundant info in problems proposal ([#808](https://github.com/microsoft/RD-Agent/issues/808)) ([76d8536](https://github.com/microsoft/RD-Agent/commit/76d8536d9ec53952383019306781d49cb3e9f75c))
-* bug fix in timer start ([#807](https://github.com/microsoft/RD-Agent/issues/807)) ([9af7161](https://github.com/microsoft/RD-Agent/commit/9af7161eb57bdd2e24b072335e9d185951c32472))
-* bug in problem identification ([#806](https://github.com/microsoft/RD-Agent/issues/806)) ([e1d5a29](https://github.com/microsoft/RD-Agent/commit/e1d5a2914046476f2f10d5884ed3c3ff956d65ff))
-* conda error information ([#941](https://github.com/microsoft/RD-Agent/issues/941)) ([fd39a94](https://github.com/microsoft/RD-Agent/commit/fd39a947763fb4a9be87b907c399bebe384df505))
-* default cost to NaN when calculation fails in LiteLLM backend ([#912](https://github.com/microsoft/RD-Agent/issues/912)) ([51a4048](https://github.com/microsoft/RD-Agent/commit/51a4048129cbfbc3b84bcf50fd8866fafb3e2da3))
-* ds trace ([#929](https://github.com/microsoft/RD-Agent/issues/929)) ([127e441](https://github.com/microsoft/RD-Agent/commit/127e441602e21a46d6313ff39133ab8ca841937e))
-* duplicate model names test in pipeline coder & runner ([#763](https://github.com/microsoft/RD-Agent/issues/763)) ([be3ee9d](https://github.com/microsoft/RD-Agent/commit/be3ee9da9882edda3c06ff7d1099d1bbda2203c3))
-* filter system metadata dirs and init missing DSTrace attribute ([#946](https://github.com/microsoft/RD-Agent/issues/946)) ([10050ef](https://github.com/microsoft/RD-Agent/commit/10050ef368ae7ec07cbf20ac4e52e21c2875eaab))
-* fix a bug in docker result extraction ([#824](https://github.com/microsoft/RD-Agent/issues/824)) ([e1c0f98](https://github.com/microsoft/RD-Agent/commit/e1c0f9826abcbc11dda215a600a2637c9ac6e984))
-* fix competition metric direction ([#784](https://github.com/microsoft/RD-Agent/issues/784)) ([3be0057](https://github.com/microsoft/RD-Agent/commit/3be0057556f46c899065ee1c7f9bafe33e79249c))
-* fix model input shape bug and costeer_model bug ([#821](https://github.com/microsoft/RD-Agent/issues/821)) ([b34bd89](https://github.com/microsoft/RD-Agent/commit/b34bd895d6d9c326aab85856a15be0cb72b2c4c8))
-* fix some minor bugs ([#758](https://github.com/microsoft/RD-Agent/issues/758)) ([963f96e](https://github.com/microsoft/RD-Agent/commit/963f96e5596bee04074135c2a0e31a8adc39ad8c))
-* fix some minor bugs in qlib scenario ([#817](https://github.com/microsoft/RD-Agent/issues/817)) ([79962a7](https://github.com/microsoft/RD-Agent/commit/79962a7ca40c77a3997a68da9ad1b5ab16728483))
-* fix the bug in the regular expression matching for stdout ([#890](https://github.com/microsoft/RD-Agent/issues/890)) ([ee57e37](https://github.com/microsoft/RD-Agent/commit/ee57e37a22af874b262c033d1606dbe7799706db))
-* fix the bug of Exceed-LLM-Context in online merge of multi-tarce ([#892](https://github.com/microsoft/RD-Agent/issues/892)) ([f760a3e](https://github.com/microsoft/RD-Agent/commit/f760a3eff7bd927a31e4958ed2f706312e83e3e3))
-* fix the problems weights bug ([#898](https://github.com/microsoft/RD-Agent/issues/898)) ([013d79f](https://github.com/microsoft/RD-Agent/commit/013d79f12060e908aeb57c3eb1bb56eea86df086))
-* fixed CI execution failures caused by document builds ([#857](https://github.com/microsoft/RD-Agent/issues/857)) ([5c116b2](https://github.com/microsoft/RD-Agent/commit/5c116b24ce727f6ed9ef39d5aa5b60442038c344))
-* get_metric_direction for aerial-cactus-identification ([#970](https://github.com/microsoft/RD-Agent/issues/970)) ([70dc62d](https://github.com/microsoft/RD-Agent/commit/70dc62de5fbd4272ecda1b6fcbcf898b3624a991))
-* import path of T ([#787](https://github.com/microsoft/RD-Agent/issues/787)) ([ac008a6](https://github.com/microsoft/RD-Agent/commit/ac008a61d03b4737ab3d994024e922839d8f3fe1))
-* improve eval alignment check (e.g. small-scale finetuning) ([#802](https://github.com/microsoft/RD-Agent/issues/802)) ([d391578](https://github.com/microsoft/RD-Agent/commit/d3915788082de640a4ce1eea6d2e607319b89c3e))
-* improve file tree and _walk symlink handling ([#877](https://github.com/microsoft/RD-Agent/issues/877)) ([516cb69](https://github.com/microsoft/RD-Agent/commit/516cb69357483ddd99f84b221a056d8491c34f9b))
-* log info ([#965](https://github.com/microsoft/RD-Agent/issues/965)) ([f1dbc21](https://github.com/microsoft/RD-Agent/commit/f1dbc2100498e22c8e5edbb2e4563c99c3d54775))
-* main bug ([#938](https://github.com/microsoft/RD-Agent/issues/938)) ([c6d34d6](https://github.com/microsoft/RD-Agent/commit/c6d34d67b8aedf5496bf6a875915ce657fc58448))
-* non-exist variable test_eval.py ([#847](https://github.com/microsoft/RD-Agent/issues/847)) ([4948c38](https://github.com/microsoft/RD-Agent/commit/4948c38560f4cf021d9354b201b22dfa5ccb9441))
-* refine feedback prompt ([#901](https://github.com/microsoft/RD-Agent/issues/901)) ([12bb2c4](https://github.com/microsoft/RD-Agent/commit/12bb2c4a1494b9aa29962905abb5e433a60eb716))
-* refine the time/memory constraints prompt in hypothesis proposal ([#856](https://github.com/microsoft/RD-Agent/issues/856)) ([51ce8ef](https://github.com/microsoft/RD-Agent/commit/51ce8ef84b4fe6590ce20599a56eee596f2f04e6))
-* Set PYTHONPATH in env.run_ret_code call in FBWorkspace class ([#755](https://github.com/microsoft/RD-Agent/issues/755)) ([68b5018](https://github.com/microsoft/RD-Agent/commit/68b501889caca754f27b57d9ab6f72184e93b15c))
-* task_gen for better understanding ([#752](https://github.com/microsoft/RD-Agent/issues/752)) ([6bfc1e5](https://github.com/microsoft/RD-Agent/commit/6bfc1e570449ee69ac110a4ced9a7cecbc0e6a73))
-* trace list but ([#852](https://github.com/microsoft/RD-Agent/issues/852)) ([32cdc57](https://github.com/microsoft/RD-Agent/commit/32cdc575bde103d71a358d4d99bd413076328ebd))
-* typo in workflow ([#861](https://github.com/microsoft/RD-Agent/issues/861)) ([0e54c9f](https://github.com/microsoft/RD-Agent/commit/0e54c9fe41d25a4cc45ab9e61bb2c2c01b854751))
-* update DS env setup with competition volume and timeout ([#878](https://github.com/microsoft/RD-Agent/issues/878)) ([816ada0](https://github.com/microsoft/RD-Agent/commit/816ada096afabe90578672b0e61b656802a30b62))
-* update feedback.py ([#772](https://github.com/microsoft/RD-Agent/issues/772)) ([133778c](https://github.com/microsoft/RD-Agent/commit/133778c67ee3349f1c2fe029bcf6a9ee14568efe))
-* update metric direction to return bool ([#791](https://github.com/microsoft/RD-Agent/issues/791)) ([0bf365e](https://github.com/microsoft/RD-Agent/commit/0bf365e7830aa86d2350b9d1c47410af46b3a7e8))
-* update runner max loop to 1 in DS scenario ([#820](https://github.com/microsoft/RD-Agent/issues/820)) ([3da378e](https://github.com/microsoft/RD-Agent/commit/3da378e986e8b776a17dbc694d29ef211192ed3e))
-* use fallback messages for missing submission and scores files ([#882](https://github.com/microsoft/RD-Agent/issues/882)) ([898fdea](https://github.com/microsoft/RD-Agent/commit/898fdeae80801d537ebc5c4a3b7df9de74c3403a))
-* use simple stdout and stderr ([#966](https://github.com/microsoft/RD-Agent/issues/966)) ([0b1c445](https://github.com/microsoft/RD-Agent/commit/0b1c445f1f0c212887ffff9f8fac44236df3607c))
-* use trace count as index ([#909](https://github.com/microsoft/RD-Agent/issues/909)) ([b87de56](https://github.com/microsoft/RD-Agent/commit/b87de56e54b206b3aada53850804474eff80b96d))
-* wrong variable test_eval.py ([#846](https://github.com/microsoft/RD-Agent/issues/846)) ([808ea6c](https://github.com/microsoft/RD-Agent/commit/808ea6cba541e60c35dd283cee9098ce46f2a59e))
-
-## [0.4.0](https://github.com/microsoft/RD-Agent/compare/v0.3.0...v0.4.0) (2025-04-04)
-
-
-### Features
-
-* (Kaggle) add base template for competition: tabular-playground-series-may-2022 ([#481](https://github.com/microsoft/RD-Agent/issues/481)) ([f3405ca](https://github.com/microsoft/RD-Agent/commit/f3405ca732eb0ddca8e18ea72f69cbd86055c4ab))
-* a unified CoSTEER to fit more scenarios ([#491](https://github.com/microsoft/RD-Agent/issues/491)) ([cddbd02](https://github.com/microsoft/RD-Agent/commit/cddbd02e3ad3ccf6ad01443777319dc5c7eb08a7))
-* add a new competition ([#474](https://github.com/microsoft/RD-Agent/issues/474)) ([2fc0d77](https://github.com/microsoft/RD-Agent/commit/2fc0d77c485a31f647e21f4578e2e326f7032964))
-* add a tool to enable saving workspace files into a specific folder ([#728](https://github.com/microsoft/RD-Agent/issues/728)) ([bca864b](https://github.com/microsoft/RD-Agent/commit/bca864b7edeafe3f88405efb695ca8acad6252f8))
-* add baseline score stat ([#590](https://github.com/microsoft/RD-Agent/issues/590)) ([2948026](https://github.com/microsoft/RD-Agent/commit/2948026c390d067b643f8c8247c1447f1dc023e4))
-* add configurable volume mode for Docker volumes in env.py ([#537](https://github.com/microsoft/RD-Agent/issues/537)) ([642a022](https://github.com/microsoft/RD-Agent/commit/642a02239431411b91959f23e69b454997ca75d5))
-* add constraint labels for semantic search ([#680](https://github.com/microsoft/RD-Agent/issues/680)) ([0584cfc](https://github.com/microsoft/RD-Agent/commit/0584cfcd13ca1a62c85390ea2ee7574370748d31))
-* add cross validation to workflow ([#700](https://github.com/microsoft/RD-Agent/issues/700)) ([82e9b00](https://github.com/microsoft/RD-Agent/commit/82e9b00be62b01673353a7aaa3ab0e2e3ecaf3ca))
-* add describe_data_folder_v2 ([#738](https://github.com/microsoft/RD-Agent/issues/738)) ([bc8e846](https://github.com/microsoft/RD-Agent/commit/bc8e8460e0246321792ff3347b1b8905416ad075))
-* add do_truncate control for the load function ([#656](https://github.com/microsoft/RD-Agent/issues/656)) ([2b960a5](https://github.com/microsoft/RD-Agent/commit/2b960a58dfdeba69522a0f72ecf0975bb6ae87ee))
-* add do_truncate control for the load function ([#656](https://github.com/microsoft/RD-Agent/issues/656)) ([2b960a5](https://github.com/microsoft/RD-Agent/commit/2b960a58dfdeba69522a0f72ecf0975bb6ae87ee))
-* add eda to data science scenario ([#639](https://github.com/microsoft/RD-Agent/issues/639)) ([35aa479](https://github.com/microsoft/RD-Agent/commit/35aa479f00edf118d43ec228e0a84c155332957a))
-* add hypothesis guidelines and rule-based ranking ([#746](https://github.com/microsoft/RD-Agent/issues/746)) ([c077b82](https://github.com/microsoft/RD-Agent/commit/c077b8239cc72904c4bc450845ed2a11aa5445f0))
-* Add line length limit to shrink_text function and settings ([#715](https://github.com/microsoft/RD-Agent/issues/715)) ([75ed5e1](https://github.com/microsoft/RD-Agent/commit/75ed5e1c2ce1bf20bb55190c10a4134e04694d2b))
-* add loop_n parameter to the main loop ([#611](https://github.com/microsoft/RD-Agent/issues/611)) ([778c166](https://github.com/microsoft/RD-Agent/commit/778c166962250e3b9e7ad85de37f62297d370b45))
-* add max time config to costeer in data science ([#645](https://github.com/microsoft/RD-Agent/issues/645)) ([534686c](https://github.com/microsoft/RD-Agent/commit/534686c2ba7d9fa979c0762ad3177c36f6d7f4cb))
-* add mlebench submission validitor ([#545](https://github.com/microsoft/RD-Agent/issues/545)) ([712d94a](https://github.com/microsoft/RD-Agent/commit/712d94a7d6f22187fc3d18bd434e71ec6997aa9f))
-* add model removal and adjust some framework logic ([#681](https://github.com/microsoft/RD-Agent/issues/681)) ([1edf881](https://github.com/microsoft/RD-Agent/commit/1edf881c63512d351c0dd074d7a1c0965ff3119b))
-* add output_path to load function of LoopBase ([#628](https://github.com/microsoft/RD-Agent/issues/628)) ([dd33726](https://github.com/microsoft/RD-Agent/commit/dd33726ac5de75dc2030d193d457d59490b3361e))
-* add pipeline coder ([#742](https://github.com/microsoft/RD-Agent/issues/742)) ([759f295](https://github.com/microsoft/RD-Agent/commit/759f295dbf1224e177006e72d694e42dd6f372b6))
-* add rank into report (mle_summary) ([#665](https://github.com/microsoft/RD-Agent/issues/665)) ([13f7922](https://github.com/microsoft/RD-Agent/commit/13f7922aaae9e4143aac4ad08ec1c556c2faf04e))
-* add restart and fix unzip ([#538](https://github.com/microsoft/RD-Agent/issues/538)) ([ed2c7d1](https://github.com/microsoft/RD-Agent/commit/ed2c7d175f1f44ca06ad7a63b08da12f6c4df9ab))
-* add retry mechanism with wait_retry decorator and refactor diff generation ([#572](https://github.com/microsoft/RD-Agent/issues/572)) ([de1cd72](https://github.com/microsoft/RD-Agent/commit/de1cd72f068ebd1e1bd5bc2ad2b12ae484d54831))
-* add the shape of the CSV to the dataset description ([#561](https://github.com/microsoft/RD-Agent/issues/561)) ([a10c881](https://github.com/microsoft/RD-Agent/commit/a10c881bd86796e6167257ad26dd165f7e46d813))
-* add timeout settings and cleanup step in data science runner ([#539](https://github.com/microsoft/RD-Agent/issues/539)) ([295abd5](https://github.com/microsoft/RD-Agent/commit/295abd56f7b58055bd27b247dfed47eb85e9b0cd))
-* add type checker to api backend & align litellm and old backend ([#647](https://github.com/microsoft/RD-Agent/issues/647)) ([d38eae9](https://github.com/microsoft/RD-Agent/commit/d38eae986a0ba69d71288fa09fcc21e227551a02))
-* align mlebench data and evaluation & several fix on kaggle workflow ([#477](https://github.com/microsoft/RD-Agent/issues/477)) ([f6c522b](https://github.com/microsoft/RD-Agent/commit/f6c522b651db3c1f6af6815347589917f46e433a))
-* **backend:** integrate LiteLLM API Backend ([#564](https://github.com/microsoft/RD-Agent/issues/564)) ([f477687](https://github.com/microsoft/RD-Agent/commit/f4776879c76a213d53875b307c94be1ea5cfd9ba))
-* base data science scenario UI ([#525](https://github.com/microsoft/RD-Agent/issues/525)) ([39917b3](https://github.com/microsoft/RD-Agent/commit/39917b354b22a8488a17396fe2245cb41e3def03))
-* condaenv & full docker env ([#668](https://github.com/microsoft/RD-Agent/issues/668)) ([084dd6d](https://github.com/microsoft/RD-Agent/commit/084dd6d748a89492ea0888acb316b9bb9efeb62f))
-* diff mode fix ([#569](https://github.com/microsoft/RD-Agent/issues/569)) ([0c509f5](https://github.com/microsoft/RD-Agent/commit/0c509f599ce19303b44d8192ec3eb634c24992d6))
-* display LLM prompt ([#676](https://github.com/microsoft/RD-Agent/issues/676)) ([8c93bba](https://github.com/microsoft/RD-Agent/commit/8c93bba82e185edcf4204cc574df5f41bcdfa9d2))
-* Dynamically find and use sample submission file in eval tests ([#542](https://github.com/microsoft/RD-Agent/issues/542)) ([5f12b44](https://github.com/microsoft/RD-Agent/commit/5f12b44c89dd26b250e914192f9beb2da38fb3ab))
-* end-to-end optimization ([#473](https://github.com/microsoft/RD-Agent/issues/473)) ([d41343a](https://github.com/microsoft/RD-Agent/commit/d41343a63d87bf3479f5ec30745ea788580495bf))
-* Enhance eval script with file cleanup and detailed submission checks ([#529](https://github.com/microsoft/RD-Agent/issues/529)) ([cf2ff92](https://github.com/microsoft/RD-Agent/commit/cf2ff9213d3a8b0fad64df7cae0c35f996d72e27))
-* exclude invalid session log folder ([#554](https://github.com/microsoft/RD-Agent/issues/554)) ([fa86e4d](https://github.com/microsoft/RD-Agent/commit/fa86e4d1805000e0e5779c662ccbb5273fda623c))
-* improve the framework's ability to adaptively adjust the model ([#629](https://github.com/microsoft/RD-Agent/issues/629)) ([93806f3](https://github.com/microsoft/RD-Agent/commit/93806f33a1e0f29a125e29303d4b984a9817c3c0))
-* independent use_azure_token_provider on chat and embedding ([#452](https://github.com/microsoft/RD-Agent/issues/452)) ([d223004](https://github.com/microsoft/RD-Agent/commit/d223004917692e231b251330cbc8676081d5a10d))
-* integrate azure deepseek r1 ([#591](https://github.com/microsoft/RD-Agent/issues/591)) ([e79ce5c](https://github.com/microsoft/RD-Agent/commit/e79ce5c38539138abe04eb9809fbde437e97bbb7))
-* kaggle refactor ([#489](https://github.com/microsoft/RD-Agent/issues/489)) ([1b057d0](https://github.com/microsoft/RD-Agent/commit/1b057d0d63a861fba4b3cb59c6c5fc1a0e3da383))
-* **kaggle:** several update in kaggle scenarios ([#476](https://github.com/microsoft/RD-Agent/issues/476)) ([245d211](https://github.com/microsoft/RD-Agent/commit/245d211dcbfb18ebcc554247a0e3a8dbecf6f3bd))
-* loader prompt &  simplify YAML loading and update data loader specifications ([#736](https://github.com/microsoft/RD-Agent/issues/736)) ([86f8bbf](https://github.com/microsoft/RD-Agent/commit/86f8bbf15895e7c198f9bc395d055ca5f02a5bb6))
-* make spec optional ([#719](https://github.com/microsoft/RD-Agent/issues/719)) ([a16b70f](https://github.com/microsoft/RD-Agent/commit/a16b70ff34c66d7e1c4c7ff5236eca8e7d8abea9))
-* Make system prompt role customizable in LLM settings ([#632](https://github.com/microsoft/RD-Agent/issues/632)) ([e4acd92](https://github.com/microsoft/RD-Agent/commit/e4acd92cc5eec6db5c29cb2d4788020fb89099b7))
-* multi log folder, replace "epxx" in workspace path ([#555](https://github.com/microsoft/RD-Agent/issues/555)) ([8a69c9c](https://github.com/microsoft/RD-Agent/commit/8a69c9c9630860c9b644356e1f71654aea222328))
-* new exp gen v2 implementation ([#725](https://github.com/microsoft/RD-Agent/issues/725)) ([5dcc2d5](https://github.com/microsoft/RD-Agent/commit/5dcc2d5fa63bbe9ae8c4817d9b40b77600440edb))
-* new-york-city-taxi-fare-prediction_template ([#488](https://github.com/microsoft/RD-Agent/issues/488)) ([a9caab7](https://github.com/microsoft/RD-Agent/commit/a9caab7bc5dc86f395a008e523355922137aef17))
-* out spec change for o1-preview ([#666](https://github.com/microsoft/RD-Agent/issues/666)) ([22894bd](https://github.com/microsoft/RD-Agent/commit/22894bdbee26b9cad73646d2975857787e515f75))
-* refactor for general data science ([#498](https://github.com/microsoft/RD-Agent/issues/498)) ([7002dc4](https://github.com/microsoft/RD-Agent/commit/7002dc4981a4f72096b438d2fe4fd9ff268c54f3))
-* refine logic for qlib_factor_from_report ([#463](https://github.com/microsoft/RD-Agent/issues/463)) ([21348d8](https://github.com/microsoft/RD-Agent/commit/21348d89e0e0eec1b4fab4e7a497f1eb34b8fe72))
-* run benchmark on gpt-4o & llama 3.1 ([#497](https://github.com/microsoft/RD-Agent/issues/497)) ([64af0b5](https://github.com/microsoft/RD-Agent/commit/64af0b5529b687cce8b5b7a1893946e15edca626))
-* summary and UI update ([#581](https://github.com/microsoft/RD-Agent/issues/581)) ([efa51f9](https://github.com/microsoft/RD-Agent/commit/efa51f9c259a06fe219f3137f0a1005e50d2bfdd))
-* template changes for some kaggle competitions ([#484](https://github.com/microsoft/RD-Agent/issues/484)) ([2e38000](https://github.com/microsoft/RD-Agent/commit/2e38000091030811fc081d72016c7bbadf7efd50))
-* track and log accumulated completion cost in LiteLLMAPIBackend ([#727](https://github.com/microsoft/RD-Agent/issues/727)) ([b294a95](https://github.com/microsoft/RD-Agent/commit/b294a95e0b7b2ef96af355cebac92d9c87f3acab))
-* update prompts and descriptions for data science components ([#731](https://github.com/microsoft/RD-Agent/issues/731)) ([c20e226](https://github.com/microsoft/RD-Agent/commit/c20e226c3e7771c9fcd1c879a8937e4694dc03eb))
-* variable printing tool of data_science coder testing ([#658](https://github.com/microsoft/RD-Agent/issues/658)) ([116c061](https://github.com/microsoft/RD-Agent/commit/116c06190b01f0b621c021726a1be23458ab1154))
-
-
-### Bug Fixes
-
-* a default conf in scen qlib ([#503](https://github.com/microsoft/RD-Agent/issues/503)) ([d64a228](https://github.com/microsoft/RD-Agent/commit/d64a228525cbedd7687c1e06132eacd0d0647697))
-* a small bug in exp_gen ([#606](https://github.com/microsoft/RD-Agent/issues/606)) ([f734dde](https://github.com/microsoft/RD-Agent/commit/f734dde0b0101e13f38151468c8ddf9e23af26ac))
-* add check when retrying gen model codes ([#699](https://github.com/microsoft/RD-Agent/issues/699)) ([3b82f15](https://github.com/microsoft/RD-Agent/commit/3b82f159474087902d3c6007d370e3282b549015))
-* add DSExperiment type check and directory validation in log proc… ([#535](https://github.com/microsoft/RD-Agent/issues/535)) ([f59b12c](https://github.com/microsoft/RD-Agent/commit/f59b12c9cc9afde82b74bc133797ff1396678627))
-* add ensemble test, change to "use cross-validation if possible" in workflow spec ([#634](https://github.com/microsoft/RD-Agent/issues/634)) ([acc97a8](https://github.com/microsoft/RD-Agent/commit/acc97a8217253497afedcfa829902b4432e1031e))
-* add force parameter for cache_with_pickle & using cache when get kaggle leaderboard ([#687](https://github.com/microsoft/RD-Agent/issues/687)) ([c8841e5](https://github.com/microsoft/RD-Agent/commit/c8841e590a925200859acba9fda4a17d4c3aa1c7))
-* add metric name check for valid scores ([#724](https://github.com/microsoft/RD-Agent/issues/724)) ([acc2ffb](https://github.com/microsoft/RD-Agent/commit/acc2ffbde4df3b53654559d14cd035ee6be6b35e))
-* add retry mechanism for GPU device check in DockerEnv ([#573](https://github.com/microsoft/RD-Agent/issues/573)) ([a780cfb](https://github.com/microsoft/RD-Agent/commit/a780cfb621dc487cc17072bfd4aedd7d581249ab))
-* add scores.csv checking in ensemble_test ([#567](https://github.com/microsoft/RD-Agent/issues/567)) ([01808b4](https://github.com/microsoft/RD-Agent/commit/01808b47c314d1daffacc0a65e0ab934a1c41d65))
-* add stdout context length setting and improve text shrinking logic ([#559](https://github.com/microsoft/RD-Agent/issues/559)) ([4ac26a6](https://github.com/microsoft/RD-Agent/commit/4ac26a65c1f18f7513480dd562566c8a96298aa7))
-* align components' name ([#701](https://github.com/microsoft/RD-Agent/issues/701)) ([295a114](https://github.com/microsoft/RD-Agent/commit/295a1148c53d00b716b2d540573a7f43e7e2d762))
-* auto continue small bug ([#598](https://github.com/microsoft/RD-Agent/issues/598)) ([75eaecf](https://github.com/microsoft/RD-Agent/commit/75eaecf36b9f70dfc2d7fedd35836acdb05f89d6))
-* avoid try-except in ensemble eval prompts ([#637](https://github.com/microsoft/RD-Agent/issues/637)) ([5c58d6e](https://github.com/microsoft/RD-Agent/commit/5c58d6e524ef848024578033ab6d47bc9b220822))
-* avoid warning for missing llama installation when not in use ([#509](https://github.com/microsoft/RD-Agent/issues/509)) ([5ec3422](https://github.com/microsoft/RD-Agent/commit/5ec342224c2c8c4cf591f1eae673e25b14218726))
-* change devault to default ([#688](https://github.com/microsoft/RD-Agent/issues/688)) ([7f401cd](https://github.com/microsoft/RD-Agent/commit/7f401cd1c3b333285acf6d6e57654f4b9f0cb6c5))
-* change ensemble test ([#622](https://github.com/microsoft/RD-Agent/issues/622)) ([5de3595](https://github.com/microsoft/RD-Agent/commit/5de35953ed0d3e2e1f4dff0e0522f2d6475079ec))
-* change summary info of log folder ([#552](https://github.com/microsoft/RD-Agent/issues/552)) ([0eb258d](https://github.com/microsoft/RD-Agent/commit/0eb258d734e9a1280a238b9a6f63eb33047ee0a7))
-* clarify an ambiguous explanation ([#705](https://github.com/microsoft/RD-Agent/issues/705)) ([5dbfc68](https://github.com/microsoft/RD-Agent/commit/5dbfc6859cbf6cc31932dae30cf05506108fc871))
-* clarify cross_validation ([#644](https://github.com/microsoft/RD-Agent/issues/644)) ([906993e](https://github.com/microsoft/RD-Agent/commit/906993ef6482f88131d1af46f5bc66a77034b549))
-* coder prompt & model test text ([#583](https://github.com/microsoft/RD-Agent/issues/583)) ([0a41227](https://github.com/microsoft/RD-Agent/commit/0a41227f267050feaeeb47ddd4d749643eb9f198))
-* correct the configuration inheritance relationship ([#671](https://github.com/microsoft/RD-Agent/issues/671)) ([30b1ff8](https://github.com/microsoft/RD-Agent/commit/30b1ff8e1ce59b741e0b81481962063014641c0b))
-* default emb model ([#702](https://github.com/microsoft/RD-Agent/issues/702)) ([4329a72](https://github.com/microsoft/RD-Agent/commit/4329a722832a201b3fa6f9d8f9d8d46f78110410))
-* direct_exp_gen to json_target_type in DSExpGen class ([#661](https://github.com/microsoft/RD-Agent/issues/661)) ([428b74a](https://github.com/microsoft/RD-Agent/commit/428b74a988157ea864ebb40e828bd9f67589c863))
-* docker error will trigger retry and data science runner loop set to 3 ([#602](https://github.com/microsoft/RD-Agent/issues/602)) ([ad785e0](https://github.com/microsoft/RD-Agent/commit/ad785e03d5db05d9191d5e772e184532835a787b))
-* ensure expected type ([#593](https://github.com/microsoft/RD-Agent/issues/593)) ([098a9a6](https://github.com/microsoft/RD-Agent/commit/098a9a6618f70fa8dd276b9014b9e7ba9621553b))
-* filter empty log traces in ds UI ([#533](https://github.com/microsoft/RD-Agent/issues/533)) ([1a2057c](https://github.com/microsoft/RD-Agent/commit/1a2057c9fc11edc4637f0baaa6dd226eb049c36e))
-* fix a bug in cross validation ([#618](https://github.com/microsoft/RD-Agent/issues/618)) ([05a4f10](https://github.com/microsoft/RD-Agent/commit/05a4f101e0b64b860ad03294619b2350004657e8))
-* fix a bug in ensemble test script ([#713](https://github.com/microsoft/RD-Agent/issues/713)) ([ad32100](https://github.com/microsoft/RD-Agent/commit/ad321000acbd9291d22fe03a9c60e57c70511c73))
-* fix a bug in initial tasks ([#635](https://github.com/microsoft/RD-Agent/issues/635)) ([edb552e](https://github.com/microsoft/RD-Agent/commit/edb552ed283119444f357fbd0b6170b2ad97712a))
-* fix a bug in kaggle conf ([#459](https://github.com/microsoft/RD-Agent/issues/459)) ([b4ed32b](https://github.com/microsoft/RD-Agent/commit/b4ed32b17ef07d8557450063765585a48d5fcd32))
-* fix a bug in progress_bar filter ([#712](https://github.com/microsoft/RD-Agent/issues/712)) ([ba5a84d](https://github.com/microsoft/RD-Agent/commit/ba5a84dee59c39cc2a8c0d428a82da1f899ce537))
-* fix a bug in proposal (add last loop's exception to last task desc) ([#596](https://github.com/microsoft/RD-Agent/issues/596)) ([419186f](https://github.com/microsoft/RD-Agent/commit/419186ffb985fe5a0aa0f7fe59c7a223e355492e))
-* fix a bug in regular expression exception processing ([#734](https://github.com/microsoft/RD-Agent/issues/734)) ([67d3702](https://github.com/microsoft/RD-Agent/commit/67d37027bbcd7294a5890a350fe16fe78e0dfa77))
-* fix a bug in threshold score display ([#592](https://github.com/microsoft/RD-Agent/issues/592)) ([0b0a2dc](https://github.com/microsoft/RD-Agent/commit/0b0a2dc512a5560a66464ad49de25d362d0dc17e))
-* fix a bug related to model_name in ensemble ([#692](https://github.com/microsoft/RD-Agent/issues/692)) ([c6ce473](https://github.com/microsoft/RD-Agent/commit/c6ce4733f32578298abe0b60f9d82611b793cc09))
-* fix a minor bug ([#694](https://github.com/microsoft/RD-Agent/issues/694)) ([1405d8d](https://github.com/microsoft/RD-Agent/commit/1405d8dafd99ecde6f3ba9dd76133d8830d03b47))
-* fix an error in model_coder prompt ([#690](https://github.com/microsoft/RD-Agent/issues/690)) ([4528826](https://github.com/microsoft/RD-Agent/commit/452882674e915dbd9e3399c26c70ce5bb86d012c))
-* fix combined_factors_df.pkl not loading in docker ([#697](https://github.com/microsoft/RD-Agent/issues/697)) ([3984b99](https://github.com/microsoft/RD-Agent/commit/3984b995aa74318b40de7712e100d4de5cc95b11))
-* fix docs build error ([#711](https://github.com/microsoft/RD-Agent/issues/711)) ([c9e1d32](https://github.com/microsoft/RD-Agent/commit/c9e1d32d6b63560350cc7cb799c3a908e2c04e42))
-* fix ExtendedSettingsConfigDict does not work ([#660](https://github.com/microsoft/RD-Agent/issues/660)) ([3a877f3](https://github.com/microsoft/RD-Agent/commit/3a877f383b908da8d027560714030b201946bb76))
-* fix kaggle templates path error ([#747](https://github.com/microsoft/RD-Agent/issues/747)) ([3b3f504](https://github.com/microsoft/RD-Agent/commit/3b3f5041514baf741fe2d4613fa651fb5d9c002d))
-* fix KeyError direct_exp_gen ([#735](https://github.com/microsoft/RD-Agent/issues/735)) ([7200682](https://github.com/microsoft/RD-Agent/commit/7200682ac4e60d3910c29a4f7c4a37b3d24e4224))
-* fix some bugs (ensemble output, HPO, model tuning) ([#648](https://github.com/microsoft/RD-Agent/issues/648)) ([818ee29](https://github.com/microsoft/RD-Agent/commit/818ee29f8e5d4765b9801463b85b42ee9516ec33))
-* fix some bugs in the ensemble component ([#595](https://github.com/microsoft/RD-Agent/issues/595)) ([c0990ab](https://github.com/microsoft/RD-Agent/commit/c0990abb06c73ae062d9a50f50cdfd6d04aded22))
-* fix some bugs in workflow unit test ([#624](https://github.com/microsoft/RD-Agent/issues/624)) ([f845dcc](https://github.com/microsoft/RD-Agent/commit/f845dcc0ee1b059b8b32485ad46bb90c7ae0fa78))
-* fix some description errors in direct_exp_gen ([#698](https://github.com/microsoft/RD-Agent/issues/698)) ([dfaacb6](https://github.com/microsoft/RD-Agent/commit/dfaacb6d06e5d5f55e950d7177570d1efebf958f))
-* fix some minor bugs and add AutoML & cross-validation ([#604](https://github.com/microsoft/RD-Agent/issues/604)) ([18c5ef2](https://github.com/microsoft/RD-Agent/commit/18c5ef268d40efe7bb9ee18aa0d250732bdda6fa))
-* fix submission file search and add TODO in env.py ([#544](https://github.com/microsoft/RD-Agent/issues/544)) ([54d930e](https://github.com/microsoft/RD-Agent/commit/54d930e91e629f0fc2f8bdd0d0d62fcad1e99a9c))
-* fix task return dict with wrong format ([#558](https://github.com/microsoft/RD-Agent/issues/558)) ([2008244](https://github.com/microsoft/RD-Agent/commit/20082440a249dd0e5a7026c2d98c9de0288dd400))
-* fix the errors in the coder and evaluator of the five components ([#576](https://github.com/microsoft/RD-Agent/issues/576)) ([c487f83](https://github.com/microsoft/RD-Agent/commit/c487f835b651cdc40b95bbbe4efcb9a617be9e40))
-* handle division by zero in percentage calculations ([#550](https://github.com/microsoft/RD-Agent/issues/550)) ([de16c91](https://github.com/microsoft/RD-Agent/commit/de16c915e1716ef8cee43ce41069ea1a09cf1f24))
-* handle invalid regex patterns in filter_progress_bar function ([#579](https://github.com/microsoft/RD-Agent/issues/579)) ([b0daee0](https://github.com/microsoft/RD-Agent/commit/b0daee0d90e193ca1d028e01c31ebf368af89601))
-* Handle ValueError when resolving relative path for uri ([#585](https://github.com/microsoft/RD-Agent/issues/585)) ([4c7765a](https://github.com/microsoft/RD-Agent/commit/4c7765a12bda5dcfd9af72b292853d9bc28c5baf))
-* include data information in cache key generation ([#566](https://github.com/microsoft/RD-Agent/issues/566)) ([26dda46](https://github.com/microsoft/RD-Agent/commit/26dda4682b7b643c164589057cb568a4d9e55e17))
-* keep some txt files ([#557](https://github.com/microsoft/RD-Agent/issues/557)) ([54aba85](https://github.com/microsoft/RD-Agent/commit/54aba851c9fa194e318d37700307df59e06c6c84))
-* mle_score save problem ([#674](https://github.com/microsoft/RD-Agent/issues/674)) ([ca2e478](https://github.com/microsoft/RD-Agent/commit/ca2e478cf25c2c8511d5f027e32f8a98fc8e3a07))
-* move docker timeout message to __run() ([#620](https://github.com/microsoft/RD-Agent/issues/620)) ([585f4f9](https://github.com/microsoft/RD-Agent/commit/585f4f96e09f70d00eb397c10bf49c09973111df))
-* move mlebench check into runner ([#556](https://github.com/microsoft/RD-Agent/issues/556)) ([b0f7965](https://github.com/microsoft/RD-Agent/commit/b0f7965f650638273710302efee2e5da037368a2))
-* move next_component_required logic to DSTrace class and accurate implement ([#612](https://github.com/microsoft/RD-Agent/issues/612)) ([c20d311](https://github.com/microsoft/RD-Agent/commit/c20d311792f33b2ccccb466c6ec3155ff8be3213))
-* patching weird azure deployment ([#494](https://github.com/microsoft/RD-Agent/issues/494)) ([89c50ae](https://github.com/microsoft/RD-Agent/commit/89c50aee2ec8bfd1cb23767ddf7dcdd023daac8b))
-* qlib and other scenario bugs ([#636](https://github.com/microsoft/RD-Agent/issues/636)) ([98de31d](https://github.com/microsoft/RD-Agent/commit/98de31d4e577c8c450c9694f73a755c19af571f7))
-* refine prompt to generate the most simple task in init stage ([#546](https://github.com/microsoft/RD-Agent/issues/546)) ([9d6feed](https://github.com/microsoft/RD-Agent/commit/9d6feed28ce034db48482d8d9741ef8c72f4bddc))
-* replace API call with build_cls_from_json_with_retry function ([#548](https://github.com/microsoft/RD-Agent/issues/548)) ([eb72a47](https://github.com/microsoft/RD-Agent/commit/eb72a47fbf9c88dacea9691b8d7e92610492d190))
-* replace func "len()" in ensemble test code to support various data type ([#739](https://github.com/microsoft/RD-Agent/issues/739)) ([ab9c7b9](https://github.com/microsoft/RD-Agent/commit/ab9c7b955f78c5de7ec08a6c1a012a76badbdd0e))
-* return 1D embedding if create_embedding receive a string input ([#670](https://github.com/microsoft/RD-Agent/issues/670)) ([4a9c318](https://github.com/microsoft/RD-Agent/commit/4a9c3180ae4a4b043b1b4a89f51ee69cb6843142))
-* rich.print error when some control char in output ([#684](https://github.com/microsoft/RD-Agent/issues/684)) ([ec0cb2a](https://github.com/microsoft/RD-Agent/commit/ec0cb2a032824023dcd04a3acc93202471d1f90a))
-* Runnable on first complete & Rename method to next_incomplete_component for clarity ([#615](https://github.com/microsoft/RD-Agent/issues/615)) ([93d9f63](https://github.com/microsoft/RD-Agent/commit/93d9f63369a78f78e1a67ab548923bb994d1d3b4))
-* runner COSTEER evaluator ([#693](https://github.com/microsoft/RD-Agent/issues/693)) ([6a379ec](https://github.com/microsoft/RD-Agent/commit/6a379ec9b84d4e4944f1e412347aae4f5a93d476))
-* save only one mle_score pkl for a running exp ([#675](https://github.com/microsoft/RD-Agent/issues/675)) ([f87ab67](https://github.com/microsoft/RD-Agent/commit/f87ab676b73cce82bd9f997ac779e31c571b53c4))
-* Set default value for 'entry' parameter in Env.run method ([#643](https://github.com/microsoft/RD-Agent/issues/643)) ([e50d242](https://github.com/microsoft/RD-Agent/commit/e50d2424b849e4181d6ca02e9cace90236665924))
-* sort file name for cache reproduction ([#588](https://github.com/microsoft/RD-Agent/issues/588)) ([7158410](https://github.com/microsoft/RD-Agent/commit/7158410fbfdd84052f9a69cf1e04e09ac07ca598))
-* sota comparison logic ([#608](https://github.com/microsoft/RD-Agent/issues/608)) ([3575372](https://github.com/microsoft/RD-Agent/commit/35753722c0800d62855faeab996d513e62cfe7de))
-* target json type & round ([#662](https://github.com/microsoft/RD-Agent/issues/662)) ([58cb58f](https://github.com/microsoft/RD-Agent/commit/58cb58f966a1db26f5ea9662a54ba12bc921ee24))
-* templates bug ([#456](https://github.com/microsoft/RD-Agent/issues/456)) ([434a868](https://github.com/microsoft/RD-Agent/commit/434a8687eeda77e27b4938fb19694c15858ee446))
-* trace summary df showing in dsapp ([#551](https://github.com/microsoft/RD-Agent/issues/551)) ([177096d](https://github.com/microsoft/RD-Agent/commit/177096d55fecb8c7dab9650ef8f5a31024cd4c1c))
-* unzip kaggle data ([#464](https://github.com/microsoft/RD-Agent/issues/464)) ([3a9fc8e](https://github.com/microsoft/RD-Agent/commit/3a9fc8e73337d3757267b6f4482499499a1b6792))
-
-## [0.3.0](https://github.com/microsoft/RD-Agent/compare/v0.2.1...v0.3.0) (2024-10-21)
-
-
-### Features
-
-* add a new template for kaggle ([#289](https://github.com/microsoft/RD-Agent/issues/289)) ([eee3ab5](https://github.com/microsoft/RD-Agent/commit/eee3ab5b25198224826cb7a8a17eab28bd5d1f7d))
-* add download submission.csv button for kaggle scenario ([#317](https://github.com/microsoft/RD-Agent/issues/317)) ([dcdcbe4](https://github.com/microsoft/RD-Agent/commit/dcdcbe46b4858bfb133ae3cca056e7f602d5cf63))
-* add kaggle command ([#271](https://github.com/microsoft/RD-Agent/issues/271)) ([0938394](https://github.com/microsoft/RD-Agent/commit/0938394b7084ffbf3294d8c23d2d34bf7322ca0b))
-* add kaggle tpl: feedback-prize ([#331](https://github.com/microsoft/RD-Agent/issues/331)) ([a288e39](https://github.com/microsoft/RD-Agent/commit/a288e399e6b0beec62729bd7d46b98a55de5ab79))
-* add more templates for kaggle ([#291](https://github.com/microsoft/RD-Agent/issues/291)) ([da752ec](https://github.com/microsoft/RD-Agent/commit/da752ec806e6f5f5679bc27ac1c072ed9a319251))
-* add normal rag into framework ([#360](https://github.com/microsoft/RD-Agent/issues/360)) ([91b0b1f](https://github.com/microsoft/RD-Agent/commit/91b0b1f66c3c1bf757cb64c4cfbdcaafe59eab74))
-* add qlib_factor_strategy ([#307](https://github.com/microsoft/RD-Agent/issues/307)) ([f8f59ff](https://github.com/microsoft/RD-Agent/commit/f8f59ff0a1be4428a68c8c27f220aabad0b6c9f0))
-* Add ranking in kaggle scenario ([#401](https://github.com/microsoft/RD-Agent/issues/401)) ([b16b4be](https://github.com/microsoft/RD-Agent/commit/b16b4beb402e0c27dfb39ee9d2a120f1b56d447c))
-* Add runtime measurement for each step and loop in RDLoop. ([#281](https://github.com/microsoft/RD-Agent/issues/281)) ([83058c8](https://github.com/microsoft/RD-Agent/commit/83058c864ceeec413dd29bf501030d5a7bd34679))
-* add s3e11 kaggle template ([#324](https://github.com/microsoft/RD-Agent/issues/324)) ([8c57524](https://github.com/microsoft/RD-Agent/commit/8c57524bead1c8f655a08763d608eb7a6dd5975e))
-* Added RepoAnalyzer to empower auto-summary of a workspace ([#264](https://github.com/microsoft/RD-Agent/issues/264)) ([0bd349a](https://github.com/microsoft/RD-Agent/commit/0bd349af50b9b881ba1774bdeb4d723529ef2aa9))
-* Added support for loading and storing RAG in Kaggle scenarios. ([#269](https://github.com/microsoft/RD-Agent/issues/269)) ([c4895de](https://github.com/microsoft/RD-Agent/commit/c4895de83f1ed000e563d42b3468a6bd9e5a4965))
-* announce Discord and WeChat ([#367](https://github.com/microsoft/RD-Agent/issues/367)) ([acac507](https://github.com/microsoft/RD-Agent/commit/acac5078a103b71afa6bd6c053b0766a6a7e609d))
-* auto submit result after one kaggle RDLoop ([#345](https://github.com/microsoft/RD-Agent/issues/345)) ([ab55d70](https://github.com/microsoft/RD-Agent/commit/ab55d7052b53a928b84dc5d5d0d2999d90ca9056))
-* better feedback & evaluation ([#346](https://github.com/microsoft/RD-Agent/issues/346)) ([cc9a8c1](https://github.com/microsoft/RD-Agent/commit/cc9a8c1eab3ca89f8c1e5de4a2bb4e7fcc0cc615))
-* Dynamic scenario based on task ([#392](https://github.com/microsoft/RD-Agent/issues/392)) ([665a037](https://github.com/microsoft/RD-Agent/commit/665a037e4fd7326c450e3fa0d0605eea26fd9ef3))
-* Factor Implement Search Enhancement ([#294](https://github.com/microsoft/RD-Agent/issues/294)) ([4ecf25f](https://github.com/microsoft/RD-Agent/commit/4ecf25f0acf2389a172b14d3dab20895daf2ab89))
-* Feature selection v3 to support all actions  ([#280](https://github.com/microsoft/RD-Agent/issues/280)) ([0047641](https://github.com/microsoft/RD-Agent/commit/00476413fbf00e36e71ab3ccb48d4e766b6ccf4d))
-* fix some bugs and add original features' description ([#259](https://github.com/microsoft/RD-Agent/issues/259)) ([1a5f45a](https://github.com/microsoft/RD-Agent/commit/1a5f45a40d821c017bdba14af8c93710707c5ea5))
-* get kaggle notebooks & disscussion text for RAG ([#371](https://github.com/microsoft/RD-Agent/issues/371)) ([cead345](https://github.com/microsoft/RD-Agent/commit/cead3450a14bf4b142ac988c27fa098c7656a95c))
-* Iceberge competition ([#372](https://github.com/microsoft/RD-Agent/issues/372)) ([c10ea4f](https://github.com/microsoft/RD-Agent/commit/c10ea4f5d4cc56a75b47cf23c7084ee189ba1a25))
-* implement isolated model feature selection loop ([#370](https://github.com/microsoft/RD-Agent/issues/370)) ([cf1292d](https://github.com/microsoft/RD-Agent/commit/cf1292de1a0153ca14ea64971e73a1c93f7d89e3))
-* Initial version if Graph RAG in KAGGLE scenario ([#301](https://github.com/microsoft/RD-Agent/issues/301)) ([fd3c0fd](https://github.com/microsoft/RD-Agent/commit/fd3c0fd26eff7d3be72fa4f2a234e33b9f796627))
-* Integrate RAG into the Kaggle scenarios. ([#262](https://github.com/microsoft/RD-Agent/issues/262)) ([be0e48a](https://github.com/microsoft/RD-Agent/commit/be0e48a7dfbee2b5d2947d09115db5db2e5266f1))
-* Kaggle loop update (Feature & Model) ([#241](https://github.com/microsoft/RD-Agent/issues/241)) ([4cf22a6](https://github.com/microsoft/RD-Agent/commit/4cf22a65c964123b4267569ee02c0c7094c54ca4))
-* kaggle templates related ([#287](https://github.com/microsoft/RD-Agent/issues/287)) ([785fdc1](https://github.com/microsoft/RD-Agent/commit/785fdc144d16fa8454b7c9d2e53e78fe7f22a29a))
-* Model context for tuning and selection ([#284](https://github.com/microsoft/RD-Agent/issues/284)) ([f2831e7](https://github.com/microsoft/RD-Agent/commit/f2831e7442510668b0ca75953b3359894803ef3c))
-* Modify FactorRowCountEvaluator and FactorIndexEvaluator to return the ratio ([#328](https://github.com/microsoft/RD-Agent/issues/328)) ([8f43f8e](https://github.com/microsoft/RD-Agent/commit/8f43f8e87a92e05b541e925910608606ec8f6c4b))
-* New competition - Optiver ([#356](https://github.com/microsoft/RD-Agent/issues/356)) ([3705efe](https://github.com/microsoft/RD-Agent/commit/3705efe3b923748655a57d76b7a236e54d361831))
-* random forest for s3e11 ([#347](https://github.com/microsoft/RD-Agent/issues/347)) ([b57846d](https://github.com/microsoft/RD-Agent/commit/b57846d29314e9a5967945d1b4895f0f48c0f5ce))
-* refine the code in model description and fix some bugs in feedback.py ([#288](https://github.com/microsoft/RD-Agent/issues/288)) ([5b124d7](https://github.com/microsoft/RD-Agent/commit/5b124d7372137e4c613eb2749ddcc773922cc7b6))
-* refine the template in several Kaggle competitions ([#343](https://github.com/microsoft/RD-Agent/issues/343)) ([034f238](https://github.com/microsoft/RD-Agent/commit/034f238ed5ec351486b21250eabc75114961936c))
-* Revise to support better hypothesis proposal  ([#390](https://github.com/microsoft/RD-Agent/issues/390)) ([c55ec0a](https://github.com/microsoft/RD-Agent/commit/c55ec0a0f577bbf7fc6228f7b87d2089ded83b31))
-* show workspace in demo ([#348](https://github.com/microsoft/RD-Agent/issues/348)) ([ddf567c](https://github.com/microsoft/RD-Agent/commit/ddf567c551b553788be022e9312c209ef6137d64))
-* support Multi output ([#330](https://github.com/microsoft/RD-Agent/issues/330)) ([3d36c45](https://github.com/microsoft/RD-Agent/commit/3d36c452ff0983800e5343834cc69f24a508ea70))
-* Supporting COVID-19 competition ([#374](https://github.com/microsoft/RD-Agent/issues/374)) ([a1b63db](https://github.com/microsoft/RD-Agent/commit/a1b63db79600edc9a74ba713c9d0be290214a592))
-* supporting Mnist competition ([#375](https://github.com/microsoft/RD-Agent/issues/375)) ([e958a34](https://github.com/microsoft/RD-Agent/commit/e958a34f5632a46ac43bff8e0d07d6ed020fdfc2))
-* Supporting Model Specifications ([#319](https://github.com/microsoft/RD-Agent/issues/319)) ([e126471](https://github.com/microsoft/RD-Agent/commit/e1264719e10b76158a91cd0ef331848e7c2de7c7))
-* supporting various Kaggle competitions & scenarios for RD-Agent ([#409](https://github.com/microsoft/RD-Agent/issues/409)) ([75eea22](https://github.com/microsoft/RD-Agent/commit/75eea22cc3d4e6f5a94c88cce915e27c507f8c50))
-* template for kaggle ([#308](https://github.com/microsoft/RD-Agent/issues/308)) ([ff97cf0](https://github.com/microsoft/RD-Agent/commit/ff97cf0155ab6941e4b5cf7d103575f934b70dc9))
-* use auto gen seed when using LLM cache ([#441](https://github.com/microsoft/RD-Agent/issues/441)) ([ca15365](https://github.com/microsoft/RD-Agent/commit/ca15365d23eeb094f42cf3dc8f5269b2f1c42bd3))
-* use unified pickle cacher & move llm config into a isolated config ([#424](https://github.com/microsoft/RD-Agent/issues/424)) ([2879ecf](https://github.com/microsoft/RD-Agent/commit/2879ecff816d97688b60909a79c7e568d42608a1))
-* xgboost gpu accelerate ([#359](https://github.com/microsoft/RD-Agent/issues/359)) ([56a5b8f](https://github.com/microsoft/RD-Agent/commit/56a5b8f9b2c6726cc64ec5b04b4ce7935d59b572))
-
-
-### Bug Fixes
-
-* a bug of developer& edit s4e8 template ([#338](https://github.com/microsoft/RD-Agent/issues/338)) ([f12ce72](https://github.com/microsoft/RD-Agent/commit/f12ce726e7de96d478a232a3c27f92439820f8b4))
-* actively raised errors aer also considered as negative feedback. ([#268](https://github.com/microsoft/RD-Agent/issues/268)) ([46ec908](https://github.com/microsoft/RD-Agent/commit/46ec908e3594ac5e4cdc4057268e2f8800f5ed1f))
-* bug of saving preprocess cache files ([#310](https://github.com/microsoft/RD-Agent/issues/310)) ([5fb0608](https://github.com/microsoft/RD-Agent/commit/5fb0608f39f113cc9807fb1f381284a0bd4da318))
-* cache ([#383](https://github.com/microsoft/RD-Agent/issues/383)) ([f2a6e75](https://github.com/microsoft/RD-Agent/commit/f2a6e75b36ca96f7733b9c2a7154ac67bd2d7c6f))
-* change css tag of kaggle competition info crawler ([#306](https://github.com/microsoft/RD-Agent/issues/306)) ([1e3d38b](https://github.com/microsoft/RD-Agent/commit/1e3d38bf1ca3654f3a90ff392ecba1dbb4e80224))
-* debug dsagent ([#387](https://github.com/microsoft/RD-Agent/issues/387)) ([8fe9511](https://github.com/microsoft/RD-Agent/commit/8fe9511e606ba148c66f384add6ab94857079541))
-* eval_method cannot catch run factor error ([#260](https://github.com/microsoft/RD-Agent/issues/260)) ([2aaab31](https://github.com/microsoft/RD-Agent/commit/2aaab317ccb7a0121063bcd85fc36c21c7b8a391))
-* fix a bug in competition metric evaluation ([#407](https://github.com/microsoft/RD-Agent/issues/407)) ([94c47d6](https://github.com/microsoft/RD-Agent/commit/94c47d6fd5c3e38fc786a83e6d0d05e8d04498f3))
-* fix a bug in mini case ([#389](https://github.com/microsoft/RD-Agent/issues/389)) ([e75bb57](https://github.com/microsoft/RD-Agent/commit/e75bb5746f63933b750406bbd34ee63c5ba76b9f))
-* fix a bug in model tuning feedback ([#316](https://github.com/microsoft/RD-Agent/issues/316)) ([8aa088d](https://github.com/microsoft/RD-Agent/commit/8aa088da2dc7525a3970c01d01987246f47d6238))
-* fix a bug in scenario.py ([#388](https://github.com/microsoft/RD-Agent/issues/388)) ([999a1eb](https://github.com/microsoft/RD-Agent/commit/999a1eb0eff9088e1b02419db741db4acf8d9ff7))
-* fix a bug in the format of the model input ([#327](https://github.com/microsoft/RD-Agent/issues/327)) ([8f0574e](https://github.com/microsoft/RD-Agent/commit/8f0574eaaadb245b8c38e09ad4821306996d926f))
-* fix a small bug in cache using module name and function name as unique folder name ([#429](https://github.com/microsoft/RD-Agent/issues/429)) ([4f8134a](https://github.com/microsoft/RD-Agent/commit/4f8134a697d952f7ac824d7ebeec64bbc4545ab3))
-* fix a typo ([#362](https://github.com/microsoft/RD-Agent/issues/362)) ([9fafabd](https://github.com/microsoft/RD-Agent/commit/9fafabdf321b818bdd2211a2324d50cd0ebe1c1f))
-* fix cache result logic ([#430](https://github.com/microsoft/RD-Agent/issues/430)) ([5e34263](https://github.com/microsoft/RD-Agent/commit/5e342637dcc862679fd0642c6ba9ef048c984845))
-* fix command injection ([#421](https://github.com/microsoft/RD-Agent/issues/421)) ([52f30a6](https://github.com/microsoft/RD-Agent/commit/52f30a6184af1295be15e855a80b84bc424fc75d))
-* fix json load error ([#386](https://github.com/microsoft/RD-Agent/issues/386)) ([bba55fb](https://github.com/microsoft/RD-Agent/commit/bba55fb48fe105f4847c1b9c476eedc80835f523))
-* fix some bugs in feedback.py and refine the prompt ([#292](https://github.com/microsoft/RD-Agent/issues/292)) ([d834052](https://github.com/microsoft/RD-Agent/commit/d8340527f133dcc649d599d90d6402eddd37859e))
-* fix some bugs in knowledge base ([#378](https://github.com/microsoft/RD-Agent/issues/378)) ([fa6ff8e](https://github.com/microsoft/RD-Agent/commit/fa6ff8e591cf1847df77d73116649c5623161573))
-* fix some bugs in rag ([#399](https://github.com/microsoft/RD-Agent/issues/399)) ([194215c](https://github.com/microsoft/RD-Agent/commit/194215c4559aee5b6ece18d65c95fb30968e2db6))
-* fix some bugs in the entire loop ([#274](https://github.com/microsoft/RD-Agent/issues/274)) ([8a564ec](https://github.com/microsoft/RD-Agent/commit/8a564ece1d87b27ee98b76db317935e802468965))
-* fix some errors in scenario.py, proposal.py and runner.py and several complex competition scenarios([#365](https://github.com/microsoft/RD-Agent/issues/365)) ([2e383b1](https://github.com/microsoft/RD-Agent/commit/2e383b175d8448a67cb470f4e3ae8977d8ec6b5b))
-* improve_execution_time_in_kaggle_loop ([#279](https://github.com/microsoft/RD-Agent/issues/279)) ([4c8f998](https://github.com/microsoft/RD-Agent/commit/4c8f998c76f1e983a5687d2c65d3251750f2a9a0))
-* kaggle data mount problem ([#297](https://github.com/microsoft/RD-Agent/issues/297)) ([795df31](https://github.com/microsoft/RD-Agent/commit/795df311e3f93cd2f3fb51ba5698adaf10f6bd62))
-* Optiver fixes ([#357](https://github.com/microsoft/RD-Agent/issues/357)) ([b054017](https://github.com/microsoft/RD-Agent/commit/b054017463af0d1784407030f2477d212118f341))
-* partial bug in bench ([#368](https://github.com/microsoft/RD-Agent/issues/368)) ([af9808f](https://github.com/microsoft/RD-Agent/commit/af9808f98736a2df07e121c2f6d7bfeb7b7d3581))
-* preprocess output format & some mistake in spelling ([#358](https://github.com/microsoft/RD-Agent/issues/358)) ([b8b2cd6](https://github.com/microsoft/RD-Agent/commit/b8b2cd6ccd3b27aa73de847e50899a8a53b71b8f))
-* rag save file ([#385](https://github.com/microsoft/RD-Agent/issues/385)) ([1cb01dd](https://github.com/microsoft/RD-Agent/commit/1cb01dd6fe595f2f5fb86487601326611dd1a57a))
-* raise error in demo when no Metric in a Loop ([#313](https://github.com/microsoft/RD-Agent/issues/313)) ([e46a78e](https://github.com/microsoft/RD-Agent/commit/e46a78eb69271cb19978aab2f3b976c2870ca082))
-* refactor Bench ([#302](https://github.com/microsoft/RD-Agent/issues/302)) ([78a87f6](https://github.com/microsoft/RD-Agent/commit/78a87f624780ff67c0fa995ae4692678a120f99c))
-* refine some codes ([#353](https://github.com/microsoft/RD-Agent/issues/353)) ([866c2e6](https://github.com/microsoft/RD-Agent/commit/866c2e63ffa3876a3d16ad37f96da41d0558b714))
-* refine the prompt ([#286](https://github.com/microsoft/RD-Agent/issues/286)) ([77966c4](https://github.com/microsoft/RD-Agent/commit/77966c4f5e9f492c437c5b4b78d89c0f875ef0d8))
-* refine the ucb algorithm ([#406](https://github.com/microsoft/RD-Agent/issues/406)) ([14f7d97](https://github.com/microsoft/RD-Agent/commit/14f7d976e03c92d6e727524e0cdad8a03b585016))
-* revert model and make SOTA model available to COSTEER ([#351](https://github.com/microsoft/RD-Agent/issues/351)) ([3b7437b](https://github.com/microsoft/RD-Agent/commit/3b7437b87e685188259779cd85a78a0b592de9de))
-* stop using markup in docker env print ([#336](https://github.com/microsoft/RD-Agent/issues/336)) ([3009889](https://github.com/microsoft/RD-Agent/commit/3009889b5e2605b5427c76f3084e0e58026bb5ae))
-* support seed and fix absolute path ([#278](https://github.com/microsoft/RD-Agent/issues/278)) ([26352e1](https://github.com/microsoft/RD-Agent/commit/26352e13121cad5be95c0de78bb9f5dda4330614))
-* template for kaggle foreset & s4e9 ([#334](https://github.com/microsoft/RD-Agent/issues/334)) ([2393a41](https://github.com/microsoft/RD-Agent/commit/2393a41e7237615ced2c3fdd5c49308236b9f276))
-* test kaggle method ([#296](https://github.com/microsoft/RD-Agent/issues/296)) ([91a6196](https://github.com/microsoft/RD-Agent/commit/91a619618be1d7db660ea2b413a78dfaba9417a1))
-* update code to fix a small bug in model cache md5 hash ([#303](https://github.com/microsoft/RD-Agent/issues/303)) ([b00e4dc](https://github.com/microsoft/RD-Agent/commit/b00e4dc2eff5b16029a2a12a6589eadac5cfd148))
-* update new feature engineering code format ([#272](https://github.com/microsoft/RD-Agent/issues/272)) ([7850b80](https://github.com/microsoft/RD-Agent/commit/7850b8006a7c89d22629b345b4f361b0f35bc60d))
-* Update prompts.yaml to constrain only one model type   ([#341](https://github.com/microsoft/RD-Agent/issues/341)) ([5b5dfee](https://github.com/microsoft/RD-Agent/commit/5b5dfeefbc7eb9dcbd9923544005c5d281262c03))
-* Update runner.py to fix a small bug ([#282](https://github.com/microsoft/RD-Agent/issues/282)) ([8aef3ab](https://github.com/microsoft/RD-Agent/commit/8aef3abcecd6002bd4bfeedcbe2c786d8bbfe2be))
-* Use fixed file name in model costeer & fixing cache ([#311](https://github.com/microsoft/RD-Agent/issues/311)) ([1f910a5](https://github.com/microsoft/RD-Agent/commit/1f910a5248bc576895ed66c2f7b2c3e046a2bc28))
-
-
-### Performance Improvements
-
-* some small upgrade to factor costeer to improve the performance ([#420](https://github.com/microsoft/RD-Agent/issues/420)) ([9eb931f](https://github.com/microsoft/RD-Agent/commit/9eb931ffd971f252380dbd33ad1db259a4f229fd))
-
-
-### Reverts
-
-* Revert feat: Factor Implement Search Enhancement ([#294](https://github.com/microsoft/RD-Agent/issues/294)) ([#305](https://github.com/microsoft/RD-Agent/issues/305)) ([f663cf4](https://github.com/microsoft/RD-Agent/commit/f663cf42a2f75cd52aef1c6b18be7c27f0641fed))
-
-## [0.2.1](https://github.com/microsoft/RD-Agent/compare/v0.2.0...v0.2.1) (2024-09-10)
-
-
-### Bug Fixes
-
-* default model value in config ([#256](https://github.com/microsoft/RD-Agent/issues/256)) ([c097585](https://github.com/microsoft/RD-Agent/commit/c097585f631f401c2c0966f6ad4c17286924f011))
-* fix_dotenv_error ([#257](https://github.com/microsoft/RD-Agent/issues/257)) ([923063c](https://github.com/microsoft/RD-Agent/commit/923063c1fd957c4ed42e97272c72b5e9545451dc))
-* readme ([#248](https://github.com/microsoft/RD-Agent/issues/248)) ([8cede22](https://github.com/microsoft/RD-Agent/commit/8cede2209922876490148459e1134da828e1fda0))
-
-## [0.2.0](https://github.com/microsoft/RD-Agent/compare/v0.1.0...v0.2.0) (2024-09-07)
-
-
-### Features
-
-* add collect info ([#233](https://github.com/microsoft/RD-Agent/issues/233)) ([89f4af9](https://github.com/microsoft/RD-Agent/commit/89f4af90fb4d95a0689bf9efc8ffd9326469c0aa))
-* add cross validation for kaggle scenario ([#236](https://github.com/microsoft/RD-Agent/issues/236)) ([e0b03ba](https://github.com/microsoft/RD-Agent/commit/e0b03ba6b5c3d9aa552b99d470e106d4e348e64d))
-* add progress status for docker env ([#215](https://github.com/microsoft/RD-Agent/issues/215)) ([538d4ef](https://github.com/microsoft/RD-Agent/commit/538d4ef2e52de795b90d3f75b2e1e877ab85c18d))
-* Added loop code for Kaggle scene. ([#211](https://github.com/microsoft/RD-Agent/issues/211)) ([975c327](https://github.com/microsoft/RD-Agent/commit/975c32715e51aec6b49537401f5fc59115e04a01))
-* Demo display effect and usage ([#162](https://github.com/microsoft/RD-Agent/issues/162)) ([8cf122a](https://github.com/microsoft/RD-Agent/commit/8cf122a0155f434fa4477ae7a6d616b5caecd3e0))
-* piloting of the framework ([#227](https://github.com/microsoft/RD-Agent/issues/227)) ([e9b103e](https://github.com/microsoft/RD-Agent/commit/e9b103e684fdd2b98cd1a89971a3fce2d6e884a1))
-* support more models for kaggle scenario ([#223](https://github.com/microsoft/RD-Agent/issues/223)) ([e3a9659](https://github.com/microsoft/RD-Agent/commit/e3a96598c0720fe092ec86d7ca8c195c7d6bcc72))
-* update model_experiment.py to support basic EDA ([#220](https://github.com/microsoft/RD-Agent/issues/220)) ([bf2684c](https://github.com/microsoft/RD-Agent/commit/bf2684c4d55ab8e1048ac0291695475ad53b0cd6))
-
-
-### Bug Fixes
-
-* fix some bugs in llm calling ([#217](https://github.com/microsoft/RD-Agent/issues/217)) ([7b010f8](https://github.com/microsoft/RD-Agent/commit/7b010f8b5940aba65a58f1d78192aa80bcd0e654))
-* package dependency. ([#234](https://github.com/microsoft/RD-Agent/issues/234)) ([46be295](https://github.com/microsoft/RD-Agent/commit/46be2952952af534fd8d98a656c704c688d7cbdd))
-* remove useless line ([#177](https://github.com/microsoft/RD-Agent/issues/177)) ([64e9a8e](https://github.com/microsoft/RD-Agent/commit/64e9a8e39a2072a962111db18f5b9565df5b0176))
-
-## [0.1.0](https://github.com/microsoft/RD-Agent/compare/v0.0.1...v0.1.0) (2024-08-09)
-
-
-### Features
-
-* add entry for rdagent. ([#187](https://github.com/microsoft/RD-Agent/issues/187)) ([121b6d9](https://github.com/microsoft/RD-Agent/commit/121b6d98de38cd03be30cbee47b40baf39a2b60b))
-* change ui entry ([#197](https://github.com/microsoft/RD-Agent/issues/197)) ([fa5d335](https://github.com/microsoft/RD-Agent/commit/fa5d3354d22240888f4fc4007d9834f7424632aa))
-* remove pdfs and enable online pdf readings ([#183](https://github.com/microsoft/RD-Agent/issues/183)) ([18c0501](https://github.com/microsoft/RD-Agent/commit/18c05016a23d694c7b12759cf1322562dcffc56a))
-
-
-### Bug Fixes
-
-* Fix a fail href in readme ([#189](https://github.com/microsoft/RD-Agent/issues/189)) ([1b89218](https://github.com/microsoft/RD-Agent/commit/1b89218f6bc697494f4a1b8a42ad18963002714f))
-* fix quick start problem ([#191](https://github.com/microsoft/RD-Agent/issues/191)) ([44f61bf](https://github.com/microsoft/RD-Agent/commit/44f61bfa1058a8efb59ca48b7f1417765aeea33e))
-* update command line in readme.md ([#192](https://github.com/microsoft/RD-Agent/issues/192)) ([9c45d24](https://github.com/microsoft/RD-Agent/commit/9c45d24a192da02f7d9765cb001097da1bc36c61))
-
-## 0.0.1 (2024-08-08)
-
-
-### Features
-
-* Add description for scenario experiments. ([#174](https://github.com/microsoft/RD-Agent/issues/174)) ([fbd8c6d](https://github.com/microsoft/RD-Agent/commit/fbd8c6d87e1424c08997103b8e8fbf264858c4ed))
-* Added QlibFactorFromReportScenario and improved the report-factor loop. ([#161](https://github.com/microsoft/RD-Agent/issues/161)) ([882c79b](https://github.com/microsoft/RD-Agent/commit/882c79bf11583980e646b130f71cfa20201ffc7b))
-* filter feature which is high correlation to former implemented features ([#145](https://github.com/microsoft/RD-Agent/issues/145)) ([e818326](https://github.com/microsoft/RD-Agent/commit/e818326422740e04a4863f7c3c18744dde2ad98f))
-* Remove redundant 'key steps' section in frontend scene display. ([#169](https://github.com/microsoft/RD-Agent/issues/169)) ([e767005](https://github.com/microsoft/RD-Agent/commit/e76700513bee29232c93b97414419df330d9be8d))
-* streamlit webapp demo for different scenarios ([#135](https://github.com/microsoft/RD-Agent/issues/135)) ([d8da7db](https://github.com/microsoft/RD-Agent/commit/d8da7db865e6653fc4740efee9a843b69bd79699))
-* Uploaded Documentation, Updated Prompts & Some Code for model demo ([#144](https://github.com/microsoft/RD-Agent/issues/144)) ([529f935](https://github.com/microsoft/RD-Agent/commit/529f935aa98623f0dc1dda29eecee3ef738dd446))
-
-
-### Bug Fixes
-
-* Add framework handling for task coding failure. ([#176](https://github.com/microsoft/RD-Agent/issues/176)) ([5e14fa5](https://github.com/microsoft/RD-Agent/commit/5e14fa54a9dd30a94aebe2643b8c9a3b85517a11))
-* Comprehensive update to factor extraction. ([#143](https://github.com/microsoft/RD-Agent/issues/143)) ([b5ea040](https://github.com/microsoft/RD-Agent/commit/b5ea04019fd5fa15c0f8b9a7e4f18f490f7057d4))
-* first round app folder cleaning ([#166](https://github.com/microsoft/RD-Agent/issues/166)) ([6a5a750](https://github.com/microsoft/RD-Agent/commit/6a5a75021912927deb5e8e4c7ad3ec4b51bfc788))
-* fix pickle problem ([#140](https://github.com/microsoft/RD-Agent/issues/140)) ([7ee4258](https://github.com/microsoft/RD-Agent/commit/7ee42587b60d94417f34332cee395cf210dc8a0e))
-* fix release CI ([#165](https://github.com/microsoft/RD-Agent/issues/165)) ([85d6a5e](https://github.com/microsoft/RD-Agent/commit/85d6a5ed91113fda34ae079b23c89aa24acd2cb2))
-* fix release CI error ([#160](https://github.com/microsoft/RD-Agent/issues/160)) ([1c9f8ef](https://github.com/microsoft/RD-Agent/commit/1c9f8ef287961731944acc9008496b4dddeddca7))
-* fix several bugs in data mining scenario ([#147](https://github.com/microsoft/RD-Agent/issues/147)) ([b233380](https://github.com/microsoft/RD-Agent/commit/b233380e2c66fb030db39424f0f040c86e37f5c4))
-* fix some small bugs in report-factor loop ([#152](https://github.com/microsoft/RD-Agent/issues/152)) ([a79f9f9](https://github.com/microsoft/RD-Agent/commit/a79f9f93406aff6305a76e6a6abd3852642e4c62))
-* fix_release_ci_error ([#150](https://github.com/microsoft/RD-Agent/issues/150)) ([4f82e99](https://github.com/microsoft/RD-Agent/commit/4f82e9960a2638af9d831581185ddd3bac5711fc))
-* Fixed some bugs introduced during refactoring. ([#167](https://github.com/microsoft/RD-Agent/issues/167)) ([f8f1445](https://github.com/microsoft/RD-Agent/commit/f8f1445283fb89aefeb2918243c35a219a51a56c))
-* optimize some prompts in factor loop. ([#158](https://github.com/microsoft/RD-Agent/issues/158)) ([c2c1330](https://github.com/microsoft/RD-Agent/commit/c2c13300b9ad315a663ec2d0eada414e56c6f54f))
-
-
-### Miscellaneous Chores
-
-* release 0.0.1 ([1feacd3](https://github.com/microsoft/RD-Agent/commit/1feacd39b21193de11e9bbecf880ddf96d7c261c))
+For earlier changes inherited from the RD-Agent project, see the [upstream changelog](https://github.com/microsoft/RD-Agent/blob/main/CHANGELOG.md).

CODE_OF_CONDUCT.md

@@ -1,9 +1,71 @@
-# Microsoft Open Source Code of Conduct
+# Contributor Covenant Code of Conduct
 
-This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+## Our Pledge
 
-Resources:
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
 
-- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
-- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
-- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+nico@predix.io.
+All complaints will be reviewed and investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

@@ -1,10 +1,10 @@
-# Contributing to RD-Agent
+# Contributing to Predix
 
-We welcome contributions and suggestions to improve RD-Agent. Whether it's solving an issue, addressing a bug, enhancing documentation, or even correcting a typo, every contribution is valuable and helps improve the project.
+We welcome contributions and suggestions to improve Predix. Whether it's solving an issue, addressing a bug, enhancing documentation, or even correcting a typo, every contribution is valuable and helps improve the project.
 
 ## Getting Started
 
-To get started, you can explore the issues list or search for `TODO:` comments in the codebase by running the command:
+To get started, you can explore the issues list or search for `TODO:` comments in the codebase by running:
 ```sh
 grep -r "TODO:"
 ```
@@ -14,7 +14,7 @@ grep -r "TODO:"
 1. **Fork the Repository**: Create a fork of the repository on GitHub.
 2. **Clone the Repository**: Clone your forked repository to your local machine.
    ```sh
-   git clone https://github.com/your-username/RD-Agent.git
+   git clone https://github.com/your-username/predix.git
    ```
 3. **Create a Branch**: Create a new branch for your changes.
    ```sh
@@ -47,4 +47,4 @@ If you encounter any issues or have suggestions for improvements, please open an
 - Update documentation as needed.
 - Test your changes thoroughly before submitting a pull request.
 
-Thank you for contributing to RD-Agent!
+Thank you for contributing to Predix!

LICENSE

@@ -1,21 +1,21 @@
-    MIT License
+MIT License
 
-    Copyright (c) Microsoft Corporation.
+Copyright (c) 2025 Predix Team
 
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
 
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
 
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Makefile

@@ -19,7 +19,7 @@ CONSTRAINTS_FILE := constraints/$(PYTHON_VERSION).txt
 PUBLIC_DIR := $(shell [ "$$READTHEDOCS" = "True" ] && echo "$$READTHEDOCS_OUTPUT/html" || echo "public")
 
 # URL and Path of changelog source code.
-CHANGELOG_URL := $(shell echo $${CI_PAGES_URL:-https://microsoft.github.io/rdagent}/_sources/changelog.md.txt)
+CHANGELOG_URL := $(shell echo $${CI_PAGES_URL:-https://predixai.github.io/predix}/_sources/changelog.md.txt)
 CHANGELOG_PATH := docs/changelog.md
 
 ########################################################################################
@@ -67,6 +67,7 @@ init-qlib-env:
 	@source $$(conda info --base)/etc/profile.d/conda.sh && conda activate qlibRDAgent && which pip && pip install pyqlib && pip install ruamel-yaml==0.17.21 && pip install torch==2.1.1 && pip install catboost==0.24.3 && conda deactivate
 
 dev:
+	$(PIPRUN) pip install -U pip setuptools wheel
 	$(PIPRUN) pip install -e .[docs,lint,package,test] -c $(CONSTRAINTS_FILE)
 	$(PIPRUN) pip install -U kaggle
 	if [ "$(CI)" != "true" ] && command -v pre-commit > /dev/null 2>&1; then pre-commit install --hook-type pre-push; fi
@@ -82,11 +83,11 @@ constraints: deepclean
 
 # Check lint with black.
 black:
-	$(PIPRUN) python -m black --check --diff . --extend-exclude test/scripts --extend-exclude git_ignore_folder -l 120
+	$(PIPRUN) python -m black --check --diff . --extend-exclude "(test/scripts|test/notebook/testfiles|git_ignore_folder|web)" -l 120
 
 # Check lint with isort.
 isort:
-	$(PIPRUN) python -m isort --check . -s git_ignore_folder -s test/scripts
+	$(PIPRUN) python -m isort --check . -s git_ignore_folder -s test/scripts -s test/notebook/testfiles -s web
 
 # Check lint with mypy.
 # First deal with the core folder, and then gradually increase the scope of detection,
@@ -98,7 +99,7 @@ mypy:
 # First deal with the core folder, and then gradually increase the scope of detection,
 # and eventually realize the detection of the complete project.
 ruff:
-	$(PIPRUN) ruff check rdagent/core --ignore FBT001,FBT002,I001   # --exclude rdagent/scripts,git_ignore_folder
+	$(PIPRUN) ruff check rdagent/core --ignore FBT001,FBT002,I001,E501   # --exclude rdagent/scripts,git_ignore_folder
 
 # Check lint with toml-sort.
 toml-sort:
@@ -119,11 +120,11 @@ pre-commit:
 
 # Auto lint with black.
 auto-black:
-	$(PIPRUN) python -m black . --extend-exclude test/scripts --extend-exclude git_ignore_folder -l 120
+	$(PIPRUN) python -m black . --extend-exclude "(test/scripts|test/notebook/testfiles|git_ignore_folder|.venv|web)" -l 120
 
 # Auto lint with isort.
 auto-isort:
-	$(PIPRUN) python -m isort . -s git_ignore_folder -s test/scripts
+	$(PIPRUN) python -m isort . -s git_ignore_folder -s test/scripts -s test/notebook/testfiles -s .venv -s web
 
 # Auto lint with toml-sort.
 auto-toml-sort:

README.md

@@ -1,422 +1,313 @@
+# Predix
+
 <h4 align="center">
-  <img src="docs/_static/logo.png" alt="RA-Agent logo" style="width:70%; ">
-  
-  <a href="https://rdagent.azurewebsites.net" target="_blank">🖥️ Live Demo</a> |
-  <a href="https://rdagent.azurewebsites.net/factor_loop" target="_blank">🎥 Demo Video</a> <a href="https://www.youtube.com/watch?v=JJ4JYO3HscM&list=PLALmKB0_N3_i52fhUmPQiL4jsO354uopR" target="_blank">▶️YouTube</a>   |
-  <a href="https://rdagent.readthedocs.io/en/latest/index.html" target="_blank">📖 Documentation</a> |
-  <a href="https://aka.ms/RD-Agent-Tech-Report" target="_blank">📄 Tech Report</a> |
-  <a href="#-paperwork-list"> 📃 Papers </a>
-</h3>
+  <strong>AI-powered Quantitative Trading Agent for EUR/USD Forex</strong>
+</h4>
 
+<p align="center">
+  <a href="#installation">Installation</a> •
+  <a href="#quick-start">Quick Start</a> •
+  <a href="#configuration">Configuration</a> •
+  <a href="#features">Features</a>
+</p>
 
-[![CI](https://github.com/microsoft/RD-Agent/actions/workflows/ci.yml/badge.svg)](https://github.com/microsoft/RD-Agent/actions/workflows/ci.yml)
-[![CodeQL](https://github.com/microsoft/RD-Agent/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/microsoft/RD-Agent/actions/workflows/github-code-scanning/codeql)
-[![Dependabot Updates](https://github.com/microsoft/RD-Agent/actions/workflows/dependabot/dependabot-updates/badge.svg)](https://github.com/microsoft/RD-Agent/actions/workflows/dependabot/dependabot-updates)
-[![Lint PR Title](https://github.com/microsoft/RD-Agent/actions/workflows/pr.yml/badge.svg)](https://github.com/microsoft/RD-Agent/actions/workflows/pr.yml)
-[![Release.yml](https://github.com/microsoft/RD-Agent/actions/workflows/release.yml/badge.svg)](https://github.com/microsoft/RD-Agent/actions/workflows/release.yml)
-[![Platform](https://img.shields.io/badge/platform-Linux-blue)](https://pypi.org/project/rdagent/#files)
-[![PyPI](https://img.shields.io/pypi/v/rdagent)](https://pypi.org/project/rdagent/)
-[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/rdagent)](https://pypi.org/project/rdagent/)
-[![Release](https://img.shields.io/github/v/release/microsoft/RD-Agent)](https://github.com/microsoft/RD-Agent/releases)
-[![GitHub](https://img.shields.io/github/license/microsoft/RD-Agent)](https://github.com/microsoft/RD-Agent/blob/main/LICENSE)
-[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
-[![Checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
-[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
-[![Chat](https://img.shields.io/badge/chat-discord-blue)](https://discord.gg/ybQ97B6Jjy)
-[![Documentation Status](https://readthedocs.org/projects/rdagent/badge/?version=latest)](https://rdagent.readthedocs.io/en/latest/?badge=latest)
-[![Readthedocs Preview](https://github.com/microsoft/RD-Agent/actions/workflows/readthedocs-preview.yml/badge.svg)](https://github.com/microsoft/RD-Agent/actions/workflows/readthedocs-preview.yml) <!-- this badge is too long, please place it in the last one to make it pretty --> 
-[![arXiv](https://img.shields.io/badge/arXiv-2505.14738-00ff00.svg)](https://arxiv.org/abs/2505.14738)
+<p align="center">
+  <a href="https://github.com/PredixAI/predix/blob/main/LICENSE"><img src="https://img.shields.io/github/license/PredixAI/predix" alt="License"></a>
+  <a href="https://pypi.org/project/predix/"><img src="https://img.shields.io/pypi/v/predix" alt="PyPI"></a>
+  <a href="https://github.com/PredixAI/predix/actions/workflows/ci.yml"><img src="https://github.com/PredixAI/predix/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+  <a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff"></a>
+</p>
 
+---
 
+## Overview
 
-# 🏆 The Best Machine Learning Engineering Agent!
+**Predix** is an autonomous AI agent for quantitative trading strategies in the EUR/USD forex market. Built on a multi-agent framework, Predix automates the full research and development cycle:
 
-[MLE-bench](https://github.com/openai/mle-bench) is a comprehensive benchmark evaluating the performance of AI agents on machine learning engineering tasks. Utilizing datasets from 75 Kaggle competitions, MLE-bench provides robust assessments of AI systems' capabilities in real-world ML engineering scenarios.
+- 📊 **Data Analysis** – Automatically analyzes market patterns and microstructure
+- 💡 **Strategy Discovery** – Proposes novel trading factors and signals
+- 🧠 **Model Evolution** – Iteratively improves predictive models
+- 📈 **Backtesting** – Validates strategies on historical 1-minute data
 
-R&D-Agent currently leads as the top-performing machine learning engineering agent on MLE-bench:
+Predix is optimized for **1-minute EUR/USD FX data** (2020–2026) and uses Qlib as the underlying backtesting engine.
 
-| Agent | Low == Lite (%) | Medium (%) | High (%) | All (%) |
-|---------|--------|-----------|---------|----------|
-| R&D-Agent o1-preview | 48.18 ± 2.49 | 8.95 ± 2.36 | 18.67 ± 2.98 | 22.4 ± 1.1 |
-| R&D-Agent o3(R)+GPT-4.1(D) | 51.52 ± 6.21 | 7.89 ± 3.33 | 16.67 ± 3.65 | 22.45 ± 2.45 |
-| AIDE o1-preview | 34.3 ± 2.4 | 8.8 ± 1.1 | 10.0 ± 1.9 | 16.9 ± 1.1 |
+---
 
-**Notes:**
-- **O3(R)+GPT-4.1(D)**: This version is designed to both reduce average time per loop and leverage a cost-effective combination of backend LLMs by seamlessly integrating Research Agent (o3) with Development Agent (GPT-4.1).
-- **AIDE o1-preview**: Represents the previously best public result on MLE-bench as reported in the original MLE-bench paper.
-- Average and standard deviation results for R&D-Agent o1-preview is based on a independent of 5 seeds and for R&D-Agent o3(R)+GPT-4.1(D) is based on 6 seeds.
-- According to MLE-Bench, the 75 competitions are categorized into three levels of complexity: **Low==Lite** if we estimate that an experienced ML engineer can produce a sensible solution in under 2 hours, excluding the time taken to train any models; **Medium** if it takes between 2 and 10 hours; and **High** if it takes more than 10 hours.
+## Installation
 
-You can inspect the detailed runs of the above results online.
-- [R&D-Agent o1-preview detailed runs](https://aka.ms/RD-Agent_MLE-Bench_O1-preview)
-- [R&D-Agent o3(R)+GPT-4.1(D) detailed runs](https://aka.ms/RD-Agent_MLE-Bench_O3_GPT41)
+### Prerequisites
 
-For running R&D-Agent on MLE-bench, refer to **[MLE-bench Guide: Running ML Engineering via MLE-bench](https://rdagent.readthedocs.io/en/latest/scens/data_science.html)**
+- **Python 3.10 or 3.11**
+- **Docker** (required for sandboxed code execution)
+- **Linux** (officially supported; macOS/Windows may work with adjustments)
 
-# 🥇 The First Data-Centric Quant Multi-Agent Framework!
+### Quick Install
 
-R&D-Agent for Quantitative Finance, in short **RD-Agent(Q)**, is the first data-centric, multi-agent framework designed to automate the full-stack research and development of quantitative strategies via coordinated factor-model co-optimization.
+```bash
+# Install from PyPI
+pip install predix
 
-![image](https://github.com/user-attachments/assets/3198bc10-47ba-4ee0-8a8e-46d5ce44f45d)
+# Or install from source
+git clone https://github.com/PredixAI/predix
+cd predix
+pip install -e .
+```
 
-Extensive experiments in real stock markets show that, at a cost under $10, RD-Agent(Q) achieves approximately 2× higher ARR than benchmark factor libraries while using over 70% fewer factors. It also surpasses state-of-the-art deep time-series models under smaller resource budgets. Its alternating factor–model optimization further delivers excellent trade-off between predictive accuracy and strategy robustness.
+### Development Setup
 
-You can learn more details about **RD-Agent(Q)** through the [paper](https://arxiv.org/abs/2505.15155) and reproduce it through the [documentation](https://rdagent.readthedocs.io/en/latest/scens/quant_agent_fin.html).
-
-# 📰 News
-| 🗞️ News        | 📝 Description                 |
-| --            | ------      |
-| [Technical Report Release](#overall-technical-report) | Overall framework description and results on MLE-bench | 
-| [R&D-Agent-Quant Release](#deep-application-in-diverse-scenarios) | Apply R&D-Agent to quant trading | 
-| MLE-Bench Results Released | R&D-Agent currently leads as the [top-performing machine learning engineering agent](#-the-best-machine-learning-engineering-agent) on MLE-bench |
-| Support LiteLLM Backend | We now fully support **[LiteLLM](https://github.com/BerriAI/litellm)** as a backend for integration with multiple LLM providers. |
-| General Data Science Agent | [Data Science Agent](https://rdagent.readthedocs.io/en/latest/scens/data_science.html) |
-| Kaggle Scenario release | We release **[Kaggle Agent](https://rdagent.readthedocs.io/en/latest/scens/data_science.html)**, try the new features!                  |
-| Official WeChat group release  | We created a WeChat group, welcome to join! (🗪[QR Code](https://github.com/microsoft/RD-Agent/issues/880)) |
-| Official Discord release  | We launch our first chatting channel in Discord (🗪[![Chat](https://img.shields.io/badge/chat-discord-blue)](https://discord.gg/ybQ97B6Jjy)) |
-| First release | **R&D-Agent** is released on GitHub |
-
-
-
-# Data Science Agent Preview
-Check out our demo video showcasing the current progress of our Data Science Agent under development:
-
-https://github.com/user-attachments/assets/3eccbecb-34a4-4c81-bce4-d3f8862f7305
-
-# 🌟 Introduction
-<div align="center">
-      <img src="docs/_static/scen.png" alt="Our focused scenario" style="width:80%; ">
-</div>
-
-R&D-Agent aims to automate the most critical and valuable aspects of the industrial R&D process, and we begin with focusing on the data-driven scenarios to streamline the development of models and data. 
-Methodologically, we have identified a framework with two key components: 'R' for proposing new ideas and 'D' for implementing them.
-We believe that the automatic evolution of R&D will lead to solutions of significant industrial value.
-
-
-<!-- Tag Cloud -->
-R&D is a very general scenario. The advent of R&D-Agent can be your
-- 💰 **Automatic Quant Factory** ([🎥Demo Video](https://rdagent.azurewebsites.net/factor_loop)|[▶️YouTube](https://www.youtube.com/watch?v=X4DK2QZKaKY&t=6s))
-- 🤖 **Data Mining Agent:** Iteratively proposing data & models ([🎥Demo Video 1](https://rdagent.azurewebsites.net/model_loop)|[▶️YouTube](https://www.youtube.com/watch?v=dm0dWL49Bc0&t=104s)) ([🎥Demo Video 2](https://rdagent.azurewebsites.net/dmm)|[▶️YouTube](https://www.youtube.com/watch?v=VIaSTZuoZg4))  and implementing them by gaining knowledge from data.
-- 🦾 **Research Copilot:** Auto read research papers ([🎥Demo Video](https://rdagent.azurewebsites.net/report_model)|[▶️YouTube](https://www.youtube.com/watch?v=BiA2SfdKQ7o)) / financial reports ([🎥Demo Video](https://rdagent.azurewebsites.net/report_factor)|[▶️YouTube](https://www.youtube.com/watch?v=ECLTXVcSx-c)) and implement model structures or building datasets.
-- 🤖 **Kaggle Agent:** Auto Model Tuning and Feature Engineering([🎥Demo Video Coming Soon...]()) and implementing them to achieve more in competitions.
-- ...
-
-You can click the links above to view the demo. We're continuously adding more methods and scenarios to the project to enhance your R&D processes and boost productivity. 
-
-Additionally, you can take a closer look at the examples in our **[🖥️ Live Demo](https://rdagent.azurewebsites.net/)**.
-
-<div align="center">
-    <a href="https://rdagent.azurewebsites.net/" target="_blank">
-        <img src="docs/_static/demo.png" alt="Watch the demo" width="80%">
-    </a>
-</div>
-
-
-# ⚡ Quick start
-
-You can try above demos by running the following command:
-
-### 🐳 Docker installation.
-Users must ensure Docker is installed before attempting most scenarios. Please refer to the [official 🐳Docker page](https://docs.docker.com/engine/install/) for installation instructions.
-Ensure the current user can run Docker commands **without using sudo**. You can verify this by executing `docker run hello-world`.
-
-### 🐍 Create a Conda Environment
-- Create a new conda environment with Python (3.10 and 3.11 are well-tested in our CI):
-  ```sh
-  conda create -n rdagent python=3.10
-  ```
-- Activate the environment:
-  ```sh
-  conda activate rdagent
-  ```
-
-### 🛠️ Install the R&D-Agent
-- You can directly install the R&D-Agent package from PyPI:
-  ```sh
-  pip install rdagent
-  ```
-
-### 💊 Health check
-- rdagent provides a health check that currently checks two things.
-  - whether the docker installation was successful.
-  - whether the default port used by the [rdagent ui](https://github.com/microsoft/RD-Agent?tab=readme-ov-file#%EF%B8%8F-monitor-the-application-results) is occupied.
-  ```sh
-  rdagent health_check
-  ```
-
-
-### ⚙️ Configuration
-- The demos requires following ability:
-  - ChatCompletion
-  - json_mode
-  - embedding query
-
-  You can set your Chat Model and Embedding Model in the following ways:
-
-- **Using LiteLLM (Default)**: We now support LiteLLM as a backend for integration with multiple LLM providers. You can configure in two ways:
-
-  **Option 1: Unified API base for both models**
-  ```bash
-  cat << EOF  > .env
-  # Set to any model supported by LiteLLM.
-  CHAT_MODEL=gpt-4o 
-  EMBEDDING_MODEL=text-embedding-3-small
-  # Configure unified API base
-  OPENAI_API_BASE=<your_unified_api_base>
-  OPENAI_API_KEY=<replace_with_your_openai_api_key>
-  ```
-
-  **Option 2: Separate API bases for Chat and Embedding models**
-  ```bash
-  cat << EOF  > .env
-  # Set to any model supported by LiteLLM.
-  # Configure separate API bases for chat and embedding
-  
-  # CHAT MODEL:
-  CHAT_MODEL=gpt-4o 
-  OPENAI_API_BASE=<your_chat_api_base>
-  OPENAI_API_KEY=<replace_with_your_openai_api_key>
-
-  # EMBEDDING MODEL:
-  # TAKE siliconflow as an example, you can use other providers.
-  # Note: embedding requires litellm_proxy prefix
-  EMBEDDING_MODEL=litellm_proxy/BAAI/bge-large-en-v1.5
-  LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
-  LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1
-  ```
-
-  Notice: If you are using reasoning models that include thought processes in their responses (such as \<think> tags), you need to set the following environment variable:
-  ```bash
-  REASONING_THINK_RM=True
-  ```
-
-- You can also use a deprecated backend if you only use `OpenAI API` or `Azure OpenAI` directly. For this deprecated setting and more configuration information, please refer to the [documentation](https://rdagent.readthedocs.io/en/latest/installation_and_configuration.html).
-
-### 🚀 Run the Application
-
-The **[🖥️ Live Demo](https://rdagent.azurewebsites.net/)** is implemented by the following commands(each item represents one demo, you can select the one you prefer):
-
-- Run the **Automated Quantitative Trading & Iterative Factors Model Joint Evolution**:  [Qlib](http://github.com/microsoft/qlib) self-loop factor & model proposal and implementation application
-  ```sh
-  rdagent fin_quant
-  ```
-
-- Run the **Automated Quantitative Trading & Iterative Factors Evolution**:  [Qlib](http://github.com/microsoft/qlib) self-loop factor proposal and implementation application
-  ```sh
-  rdagent fin_factor
-  ```
-
-- Run the **Automated Quantitative Trading & Iterative Model Evolution**: [Qlib](http://github.com/microsoft/qlib) self-loop model proposal and implementation application
-  ```sh
-  rdagent fin_model
-  ```
+```bash
+# Create conda environment
+conda create -n predix python=3.10
+conda activate predix
 
-- Run the **Automated Quantitative Trading & Factors Extraction from Financial Reports**:  Run the [Qlib](http://github.com/microsoft/qlib) factor extraction and implementation application based on financial reports
-  ```sh
-  # 1. Generally, you can run this scenario using the following command:
-  rdagent fin_factor_report --report_folder=<Your financial reports folder path>
-
-  # 2. Specifically, you need to prepare some financial reports first. You can follow this concrete example:
-  wget https://github.com/SunsetWolf/rdagent_resource/releases/download/reports/all_reports.zip
-  unzip all_reports.zip -d git_ignore_folder/reports
-  rdagent fin_factor_report --report_folder=git_ignore_folder/reports
-  ```
-
-- Run the **Automated Model Research & Development Copilot**: model extraction and implementation application
-  ```sh
-  # 1. Generally, you can run your own papers/reports with the following command:
-  rdagent general_model <Your paper URL>
+# Install in editable mode with dev dependencies
+make dev
+```
 
-  # 2. Specifically, you can do it like this. For more details and additional paper examples, use `rdagent general_model -h`:
-  rdagent general_model  "https://arxiv.org/pdf/2210.09789"
-  ```
+---
 
-- Run the **Automated Kaggle Model Tuning & Feature Engineering**:  self-loop model proposal and feature engineering implementation application <br />
-  > Using **sf-crime** *(San Francisco Crime Classification)* as an example. <br />
-  > 1. Register and login on the [Kaggle](https://www.kaggle.com/) website. <br />
-  > 2. Configuring the Kaggle API. <br />
-  > (1) Click on the avatar (usually in the top right corner of the page) -> `Settings` -> `Create New Token`, A file called `kaggle.json` will be downloaded. <br />
-  > (2) Move `kaggle.json` to `~/.config/kaggle/` <br />
-  > (3) Modify the permissions of the kaggle.json file. Reference command: `chmod 600 ~/.config/kaggle/kaggle.json` <br />
-  > 3. Join the competition: Click `Join the competition` -> `I Understand and Accept` at the bottom of the [competition details page](https://www.kaggle.com/competitions/sf-crime/data).
-  ```bash
-  # Generally, you can run the Kaggle competition program with the following command:
-  rdagent data_science --competition <your competition name>
+## Quick Start
 
-  # Specifically, you need to create a folder for storing competition files (e.g., competition description file, competition datasets, etc.), and configure the path to the folder in your environment. In addition, you need to use chromedriver when you download the competition descriptors, which you can follow for this specific example:
-  
-  # 1. Install chromedriver.
+### 1. Health Check
 
-  # 2. Add the competition description file path to the `.env` file.
-  mkdir -p ./git_ignore_folder/kaggle_data
-  dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/kaggle_data"
-  dotenv set DS_IF_USING_MLE_DATA True
+Verify your environment is properly configured:
 
-  # 3. run the application
-  rdagent data_science --competition sf-crime
-  ```
-
-### 🖥️ Monitor the Application Results
-- You can run the following command for our demo program to see the run logs.
+```bash
+rdagent health_check --no-check-env
+```
 
-  ```sh
-  rdagent ui --port 19899 --log_dir <your log folder like "log/">
-  ```
+### 2. Configure LLM Backend
 
-  **Note:** Although port 19899 is not commonly used, but before you run this demo, you need to check if port 19899 is occupied. If it is, please change it to another port that is not occupied.
+Create a `.env` file in your project root:
 
-  You can check if a port is occupied by running the following command.
+```bash
+# Example: OpenAI configuration
+cat << EOF > .env
+CHAT_MODEL=gpt-4o
+EMBEDDING_MODEL=text-embedding-3-small
+OPENAI_API_BASE=https://api.openai.com/v1
+OPENAI_API_KEY=your-api-key-here
+EOF
+```
 
-  ```sh
-  rdagent health_check
-  ```
+**Alternative providers:**
 
-# 🏭 Scenarios
+- **Azure OpenAI**: Set `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`
+- **DeepSeek**: Use `CHAT_MODEL=deepseek/deepseek-chat` with `DEEPSEEK_API_KEY`
+- **SiliconFlow (embedding)**: Use `EMBEDDING_MODEL=litellm_proxy/BAAI/bge-m3`
 
-We have applied R&D-Agent to multiple valuable data-driven industrial scenarios.
+### 3. Run Quantitative Trading Loop
 
+```bash
+# Full factor & model co-evolution
+rdagent fin_quant
 
-## 🎯 Goal: Agent for Data-driven R&D
+# Factor-only evolution
+rdagent fin_factor
 
-In this project, we are aiming to build an Agent to automate Data-Driven R\&D that can
-+ 📄 Read real-world material (reports, papers, etc.) and **extract** key formulas, descriptions of interested **features** and **models**, which are the key components of data-driven R&D .
-+ 🛠️ **Implement** the extracted formulas (e.g., features, factors, and models) in runnable codes.
-   + Due to the limited ability of LLM in implementing at once, build an evolving process for the agent to improve performance by learning from feedback and knowledge.
-+ 💡 Propose **new ideas** based on current knowledge and observations.
+# Model-only evolution
+rdagent fin_model
+```
 
-<!-- ![Data-Centric R&D Overview](docs/_static/overview.png) -->
+### 4. Monitor Results
 
-## 📈 Scenarios/Demos
+```bash
+# Start the UI dashboard
+rdagent ui --port 19899 --log-dir log/ --data-science
+```
 
-In the two key areas of data-driven scenarios, model implementation and data building, our system aims to serve two main roles: 🦾Copilot and 🤖Agent. 
-- The 🦾Copilot follows human instructions to automate repetitive tasks. 
-- The 🤖Agent, being more autonomous, actively proposes ideas for better results in the future.
+Then open `http://127.0.0.1:19899` in your browser.
 
-The supported scenarios are listed below:
+---
 
-| Scenario/Target | Model Implementation                   | Data Building                                                                      |
-| --              | --                                     | --                                                                                 |
-| **💹 Finance**      | 🤖 [Iteratively Proposing Ideas & Evolving](https://rdagent.azurewebsites.net/model_loop)[▶️YouTube](https://www.youtube.com/watch?v=dm0dWL49Bc0&t=104s) |  🤖 [Iteratively Proposing Ideas & Evolving](https://rdagent.azurewebsites.net/factor_loop) [▶️YouTube](https://www.youtube.com/watch?v=X4DK2QZKaKY&t=6s) <br/>   🦾 [Auto reports reading & implementation](https://rdagent.azurewebsites.net/report_factor)[▶️YouTube](https://www.youtube.com/watch?v=ECLTXVcSx-c)  |
-| **🩺 Medical**      | 🤖 [Iteratively Proposing Ideas & Evolving](https://rdagent.azurewebsites.net/dmm)[▶️YouTube](https://www.youtube.com/watch?v=VIaSTZuoZg4) | -                                                                                  |
-| **🏭 General**      | 🦾 [Auto paper reading & implementation](https://rdagent.azurewebsites.net/report_model)[▶️YouTube](https://www.youtube.com/watch?v=BiA2SfdKQ7o) <br/> 🤖 Auto Kaggle Model Tuning   | 🤖Auto Kaggle feature Engineering |
+## Configuration
 
-- **[RoadMap](https://rdagent.readthedocs.io/en/latest/scens/data_science.html#roadmap)**: Currently, we are working hard to add new features to the Kaggle scenario.
+### Data Configuration
 
-Different scenarios vary in entrance and configuration. Please check the detailed setup tutorial in the scenarios documents.
+Edit [`data_config.yaml`](data_config.yaml) to customize:
 
-Here is a gallery of [successful explorations](https://github.com/SunsetWolf/rdagent_resource/releases/download/demo_traces/demo_traces.zip) (5 traces showed in **[🖥️ Live Demo](https://rdagent.azurewebsites.net/)**). You can download and view the execution trace using [this command](https://github.com/microsoft/RD-Agent?tab=readme-ov-file#%EF%B8%8F-monitor-the-application-results) from the documentation.
+```yaml
+instrument: EURUSD
+frequency: 1min
+data_path: ~/.qlib/qlib_data/eurusd_1min_data
 
-Please refer to **[📖readthedocs_scen](https://rdagent.readthedocs.io/en/latest/scens/catalog.html)** for more details of the scenarios.
+# Walk-forward split
+train_start: "2022-03-14"
+train_end:   "2024-06-30"
+valid_start: "2024-07-01"
+valid_end:   "2024-12-31"
+test_start:  "2025-01-01"
+test_end:    "2026-03-20"
 
-# ⚙️ Framework
+# Market context for LLM prompts
+market_context:
+  spread_bps: 1.5
+  target_arr: 9.62          # Target annual return (%)
+  max_drawdown: 20          # Max drawdown (%)
+```
 
-<div align="center">
-    <img src="docs/_static/Framework-RDAgent.png" alt="Framework-RDAgent" width="85%">
-</div>
+### Environment Variables
 
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `CHAT_MODEL` | LLM for reasoning | `gpt-4o`, `deepseek-chat` |
+| `EMBEDDING_MODEL` | Embedding model | `text-embedding-3-small` |
+| `OPENAI_API_KEY` | API key for OpenAI | `sk-...` |
+| `DEEPSEEK_API_KEY` | API key for DeepSeek | `sk-...` |
+| `DS_LOCAL_DATA_PATH` | Local data directory | `./data` |
 
-Automating the R&D process in data science is a highly valuable yet underexplored area in industry. We propose a framework to push the boundaries of this important research field.
+---
 
-The research questions within this framework can be divided into three main categories:
-| Research Area | Paper/Work List |
-|--------------------|-----------------|
-| **Benchmark the R&D abilities** | [Benchmark](#benchmark) |
-| **Idea proposal:** Explore new ideas or refine existing ones | [Research](#research) |
-| **Ability to realize ideas:** Implement and execute ideas | [Development](#development) |
+## Features
 
-We believe that the key to delivering high-quality solutions lies in the ability to evolve R&D capabilities. Agents should learn like human experts, continuously improving their R&D skills.
+### 🔄 Iterative Factor Evolution
 
-More documents can be found in the **[📖 readthedocs](https://rdagent.readthedocs.io/)**.
+Predix continuously proposes, implements, and validates new alpha factors:
 
-# 📃 Paper/Work list
+- Learns from backtest feedback
+- Avoids overfitting through walk-forward validation
+- Discovers non-obvious patterns in order flow, volatility, and session dynamics
 
-## Overall Technical Report
-- [R&D-Agent: Automating Data-Driven AI Solution Building Through LLM-Powered Automated Research, Development, and Evolution](https://arxiv.org/abs/2505.14738)
-```BibTeX
-@misc{yang2024rdagent,
-    title={R\&D-Agent: Automating Data-Driven AI Solution Building Through LLM-Powered Automated Research, Development, and Evolution},
-    author={Xu Yang and Xiao Yang and Shikai Fang and Bowen Xian and Yuante Li and Jian Wang and Minrui Xu and Haoran Pan and Xinpeng Hong and Weiqing Liu and Yelong Shen and Weizhu Chen and Jiang Bian},
-    year={2025},
-    eprint={2505.14738},
-    archivePrefix={arXiv},
-    primaryClass={cs.AI},
-    url={https://arxiv.org/abs/2505.14738}
+### 🧠 Model Architecture Search
+
+Automatically explores and refines predictive models:
+
+- Linear baselines (LightGBM, XGBoost)
+- Deep learning (LSTM, Transformer, Temporal CNN)
+- Ensemble methods
+
+### 📚 Knowledge Base
+
+Built-in knowledge accumulation across loops:
+
+- Successful factors are archived
+- Failed attempts inform future proposals
+- Cross-loop learning improves robustness
+
+### 🖥️ Interactive UI
+
+Real-time dashboard for monitoring:
+
+- Factor performance metrics
+- Model architecture evolution
+- Cumulative returns and drawdowns
+- Code diffs and implementation history
+
+---
+
+## Project Structure
+
+```
+predix/
+├── rdagent/                 # Core agent framework
+│   ├── app/                 # CLI and scenario apps
+│   ├── components/          # Reusable agent components
+│   ├── core/                # Core abstractions
+│   ├── scenarios/           # Domain-specific scenarios
+│   └── utils/               # Utilities
+├── constraints/             # Constraint definitions
+├── docs/                    # Documentation
+├── web/                     # Web UI frontend
+├── data_config.yaml         # Data configuration
+├── pyproject.toml           # Project metadata
+└── requirements.txt         # Dependencies
+```
+
+---
+
+## Data Setup
+
+Predix uses 1-minute EUR/USD data. To prepare your dataset:
+
+```bash
+# Run the data setup script (if provided)
+./setup_predix_eurusd.sh
+
+# Or manually place data in:
+# ~/.qlib/qlib_data/eurusd_1min_data/
+```
+
+Expected data columns: `$open`, `$close`, `$high`, `$low`, `$volume`
+
+---
+
+## CLI Commands
+
+| Command | Description |
+|---------|-------------|
+| `rdagent fin_quant` | Full factor & model co-evolution |
+| `rdagent fin_factor` | Factor-only evolution |
+| `rdagent fin_model` | Model-only evolution |
+| `rdagent fin_factor_report --report-folder=<path>` | Extract factors from financial reports |
+| `rdagent general_model <paper-url>` | Extract model from research paper |
+| `rdagent data_science --competition <name>` | Kaggle/data science competition mode |
+| `rdagent ui --port 19899 --log-dir <path>` | Start monitoring dashboard |
+| `rdagent health_check` | Validate environment setup |
+
+---
+
+## Requirements
+
+Core dependencies (see [`requirements.txt`](requirements.txt) for full list):
+
+- **LLM**: `openai`, `litellm`
+- **Data**: `pandas`, `numpy`, `pyarrow`
+- **ML**: `scikit-learn`, `lightgbm`, `xgboost`
+- **Backtesting**: `qlib` (via Docker)
+- **UI**: `streamlit`, `plotly`, `flask`
+
+---
+
+## License
+
+This project is licensed under the **MIT License** – see the [`LICENSE`](LICENSE) file for details.
+
+---
+
+## Contributing
+
+Contributions are welcome! Please:
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+For major changes, please open an issue first to discuss your approach.
+
+---
+
+## Citation
+
+If you use Predix in your research, please cite the underlying framework:
+
+```bibtex
+@misc{yang2025rdagentllmagentframeworkautonomous,
+  title={R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science},
+  author={Yang, Xu and Yang, Xiao and Fang, Shikai and Zhang, Yifei and Wang, Jian and Xian, Bowen and Li, Qizheng and Li, Jingyuan and Xu, Minrui and Li, Yuante and others},
+  year={2025},
+  eprint={2505.14738},
+  archivePrefix={arXiv},
+  primaryClass={cs.AI}
 }
 ```
-![image](https://github.com/user-attachments/assets/28b0488d-a546-4fef-8dc5-563ed64a9b4d)
 
-## 📊 Benchmark
-- [Towards Data-Centric Automatic R&D](https://arxiv.org/abs/2404.11276)
-```BibTeX
-@misc{chen2024datacentric,
-    title={Towards Data-Centric Automatic R&D},
-    author={Haotian Chen and Xinjie Shen and Zeqi Ye and Wenjun Feng and Haoxue Wang and Xiao Yang and Xu Yang and Weiqing Liu and Jiang Bian},
-    year={2024},
-    eprint={2404.11276},
-    archivePrefix={arXiv},
-    primaryClass={cs.AI}
-}
-```
-![image](https://github.com/user-attachments/assets/494f55d3-de9e-4e73-ba3d-a787e8f9e841)
+---
 
-## 🔍 Research
+## Support
 
-In a data mining expert's daily research and development process, they propose a hypothesis (e.g., a model structure like RNN can capture patterns in time-series data), design experiments (e.g., finance data contains time-series and we can verify the hypothesis in this scenario), implement the experiment as code (e.g., Pytorch model structure), and then execute the code to get feedback (e.g., metrics, loss curve, etc.). The experts learn from the feedback and improve in the next iteration.
+- **Issues**: [GitHub Issues](https://github.com/PredixAI/predix/issues)
+- **Documentation**: [Read the Docs](https://rdagent.readthedocs.io/)
 
-Based on the principles above, we have established a basic method framework that continuously proposes hypotheses, verifies them, and gets feedback from the real-world practice. This is the first scientific research automation framework that supports linking with real-world verification.
+---
 
-For more detail, please refer to our **[🖥️ Live Demo page](https://rdagent.azurewebsites.net)**.
+## Disclaimer
 
-## 🛠️ Development
+Predix is provided "as is" for **research and educational purposes only**. It is **not** intended for:
 
-- [Collaborative Evolving Strategy for Automatic Data-Centric Development](https://arxiv.org/abs/2407.18690)
-```BibTeX
-@misc{yang2024collaborative,
-    title={Collaborative Evolving Strategy for Automatic Data-Centric Development},
-    author={Xu Yang and Haotian Chen and Wenjun Feng and Haoxue Wang and Zeqi Ye and Xinjie Shen and Xiao Yang and Shizhao Sun and Weiqing Liu and Jiang Bian},
-    year={2024},
-    eprint={2407.18690},
-    archivePrefix={arXiv},
-    primaryClass={cs.AI}
-}
-```
-![image](https://github.com/user-attachments/assets/75d9769b-0edd-4caf-9d45-57d1e577054b)
+- Live trading or financial advice
+- Production use without thorough testing
+- Replacement of qualified financial professionals
 
-## Deep Application in Diverse Scenarios
-
-- [R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization](https://arxiv.org/abs/2505.15155)
-```BibTeX
-@misc{li2025rdagentquant,
-    title={R\&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization},
-    author={Yuante Li and Xu Yang and Xiao Yang and Minrui Xu and Xisen Wang and Weiqing Liu and Jiang Bian},
-    year={2025},
-    eprint={2505.15155},
-    archivePrefix={arXiv},
-    primaryClass={cs.AI}
-}
-```
-![image](https://github.com/user-attachments/assets/3186f67a-c2f8-4b6b-8bb9-a9b959c13866)
-
-
-# 🤝 Contributing
-
-We welcome contributions and suggestions to improve R&D-Agent. Please refer to the [Contributing Guide](CONTRIBUTING.md) for more details on how to contribute.
-
-Before submitting a pull request, ensure that your code passes the automatic CI checks.
-
-## 📝 Guidelines
-This project welcomes contributions and suggestions.
-Contributing to this project is straightforward and rewarding. Whether it's solving an issue, addressing a bug, enhancing documentation, or even correcting a typo, every contribution is valuable and helps improve R&D-Agent.
-
-To get started, you can explore the issues list, or search for `TODO:` comments in the codebase by running the command `grep -r "TODO:"`.
-
-<img src="https://img.shields.io/github/contributors-anon/microsoft/RD-Agent"/>
-
-<a href="https://github.com/microsoft/RD-Agent/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=microsoft/RD-Agent&max=100&columns=15" />
-</a>
-
-Before we released R&D-Agent as an open-source project on GitHub, it was an internal project within our group. Unfortunately, the internal commit history was not preserved when we removed some confidential code. As a result, some contributions from our group members, including Haotian Chen, Wenjun Feng, Haoxue Wang, Zeqi Ye, Xinjie Shen, and Jinhui Li, were not included in the public commits.
-
-# ⚖️ Legal disclaimer
-<p style="line-height: 1; font-style: italic;">The RD-agent is provided “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. The RD-agent is aimed to facilitate research and development process in the financial industry and not ready-to-use for any financial investment or advice. Users shall independently assess and test the risks of the RD-agent in a specific use scenario, ensure the responsible use of AI technology, including but not limited to developing and integrating risk mitigation measures, and comply with all applicable laws and regulations in all applicable jurisdictions. The RD-agent does not provide financial opinions or reflect the opinions of Microsoft, nor is it designed to replace the role of qualified financial professionals in formulating, assessing, and approving finance products. The inputs and outputs of the RD-agent belong to the users and users shall assume all liability under any theory of liability, whether in contract, torts, regulatory, negligence, products liability, or otherwise, associated with use of the RD-agent and any inputs and outputs thereof.</p>
+Users assume all liability and should comply with applicable laws and regulations in their jurisdiction. Past performance does not guarantee future results.

SECURITY.md

@@ -1,41 +1,20 @@
-<!-- BEGIN MICROSOFT SECURITY.MD V0.0.9 BLOCK -->
+# Security Policy
 
-## Security
+## Reporting a Vulnerability
 
-Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet) and [Xamarin](https://github.com/xamarin).
-
-If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/security.md/definition), please report it to us as described below.
-
-## Reporting Security Issues
+We take the security of Predix seriously. If you believe you have found a security vulnerability, please report it to us as described below.
 
 **Please do not report security vulnerabilities through public GitHub issues.**
 
-Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/security.md/msrc/create-report).
+Instead, please report them via email to:
+- **Email**: nico@predix.io
 
-If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/security.md/msrc/pgp).
-
-You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 
-
-Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
-
-  * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
-  * Full paths of source file(s) related to the manifestation of the issue
-  * The location of the affected source code (tag/branch/commit or direct URL)
-  * Any special configuration required to reproduce the issue
-  * Step-by-step instructions to reproduce the issue
-  * Proof-of-concept or exploit code (if possible)
-  * Impact of the issue, including how an attacker might exploit the issue
-
-This information will help us triage your report more quickly.
-
-If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/security.md/msrc/bounty) page for more details about our active programs.
+You should receive a response within 48 hours. If for some reason you do not, please follow up via email to ensure we received your original message.
 
 ## Preferred Languages
 
 We prefer all communications to be in English.
 
-## Policy
+## Security Updates
 
-Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/security.md/cvd).
-
-<!-- END MICROSOFT SECURITY.MD BLOCK -->
+Security updates will be released as patch versions. Please ensure you are using the latest version of Predix to benefit from security fixes.

SUPPORT.md

@@ -1,25 +1,25 @@
-# TODO: The maintainer of this repo has not yet edited this file
-
-**REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project?
-
-- **No CSS support:** Fill out this template with information about how to file issues and get help.
-- **Yes CSS support:** Fill out an intake form at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). CSS will work with/help you to determine next steps.
-- **Not sure?** Fill out an intake as though the answer were "Yes". CSS will help you decide.
-
-*Then remove this first heading from this SUPPORT.MD file before publishing your repo.*
-
-# Support
-
-## How to file issues and get help  
-
-This project uses GitHub Issues to track bugs and feature requests. Please search the existing 
-issues before filing new issues to avoid duplicates.  For new issues, file your bug or 
-feature request as a new Issue.
-
-For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE 
-FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER
-CHANNEL. WHERE WILL YOU HELP PEOPLE?**.
-
-## Microsoft Support Policy  
-
-Support for this **PROJECT or PRODUCT** is limited to the resources listed above.
+# Support
+
+## How to file issues and get help
+
+This project uses GitHub Issues to track bugs and feature requests. Please search the existing
+issues before filing new issues to avoid duplicates. For new issues, file your bug or
+feature request as a new Issue.
+
+- **Issues**: [https://github.com/PredixAI/predix/issues](https://github.com/PredixAI/predix/issues)
+
+For help and questions about using this project, please reach out via:
+
+- **Email**: nico@predix.io
+- **GitHub Discussions**: [https://github.com/PredixAI/predix/discussions](https://github.com/PredixAI/predix/discussions)
+
+## Community Support
+
+We encourage users to help each other through GitHub Discussions or by contributing
+answers to issues. If you find a solution to a problem, please consider sharing it
+publicly to help others.
+
+## Support Policy
+
+Support is provided on a best-effort basis by the maintainers and community.
+For critical issues or commercial support needs, please contact the maintainers directly.

TODO.md

@@ -1,10 +1,16 @@
-We encourage to set the TODOs in code. But some TODOs are more global.
-So we place it here.
+# TODOs
 
+This file tracks global TODOs for the Predix project. Individual TODOs in code should be addressed in their respective modules.
 
-- [ ] Aligning the naming of files in components & scenarios.
-  - We would like to have the same logic for naming convention in components(reusable components for all scenarios) and scenarios (componets for specific scenario).
-  - But now we have following mismatch
-    - `coder` in `components` & `developer` in `components`
-- [ ] The name of the folders mismatch with the content in them.
-  - Why are scenarios in experiments?
+## Current Global TODOs
+
+- [ ] Align naming conventions for files in `components/` and `scenarios/` directories
+  - Current mismatch: `coder` in `components/` vs `developer` in `scenarios/`
+- [ ] Clean up folder naming inconsistencies
+  - Investigate why some scenario-related code is in `experiments/` folders
+
+## Completed
+
+- [x] Rebrand from RD-Agent to Predix for EUR/USD focus
+- [x] Remove Microsoft-specific references
+- [x] Update documentation for PredixAI organization

apply_config.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""
+Liest data_config.yaml und schreibt alle Werte in:
+- .env (Zeiträume, Pfade)
+- generate.py (Qlib Datengenerierung)
+"""
+import yaml
+import re
+from pathlib import Path
+
+CONFIG = Path(__file__).parent / "data_config.yaml"
+ENV = Path(__file__).parent / ".env"
+GENERATE = Path("/home/nico/miniconda3/envs/rdagent/lib/python3.10/site-packages/rdagent/scenarios/qlib/experiment/factor_data_template/generate.py")
+
+with open(CONFIG) as f:
+    cfg = yaml.safe_load(f)
+
+# --- .env updaten ---
+env_text = ENV.read_text()
+
+replacements = {
+    r"QLIB_DATA_DIR=.*": f"QLIB_DATA_DIR={cfg['data_path'].replace('~', str(Path.home()))}",
+    r"QLIB_FREQ=.*": f"QLIB_FREQ={cfg['frequency']}",
+    r"QLIB_FACTOR_TRAIN_START=.*": f"QLIB_FACTOR_TRAIN_START={cfg['train_start']}",
+    r"QLIB_FACTOR_TRAIN_END=.*": f"QLIB_FACTOR_TRAIN_END={cfg['train_end']}",
+    r"QLIB_FACTOR_VALID_START=.*": f"QLIB_FACTOR_VALID_START={cfg['valid_start']}",
+    r"QLIB_FACTOR_VALID_END=.*": f"QLIB_FACTOR_VALID_END={cfg['valid_end']}",
+    r"QLIB_FACTOR_TEST_START=.*": f"QLIB_FACTOR_TEST_START={cfg['test_start']}",
+    r"QLIB_FACTOR_TEST_END=.*": f"QLIB_FACTOR_TEST_END={cfg['test_end']}",
+    r"QLIB_MODEL_TRAIN_START=.*": f"QLIB_MODEL_TRAIN_START={cfg['train_start']}",
+    r"QLIB_MODEL_TRAIN_END=.*": f"QLIB_MODEL_TRAIN_END={cfg['train_end']}",
+    r"QLIB_MODEL_VALID_START=.*": f"QLIB_MODEL_VALID_START={cfg['valid_start']}",
+    r"QLIB_MODEL_VALID_END=.*": f"QLIB_MODEL_VALID_END={cfg['valid_end']}",
+    r"QLIB_MODEL_TEST_START=.*": f"QLIB_MODEL_TEST_START={cfg['test_start']}",
+    r"QLIB_MODEL_TEST_END=.*": f"QLIB_MODEL_TEST_END={cfg['test_end']}",
+    r"QLIB_QUANT_TRAIN_START=.*": f"QLIB_QUANT_TRAIN_START={cfg['train_start']}",
+    r"QLIB_QUANT_TRAIN_END=.*": f"QLIB_QUANT_TRAIN_END={cfg['train_end']}",
+    r"QLIB_QUANT_VALID_START=.*": f"QLIB_QUANT_VALID_START={cfg['valid_start']}",
+    r"QLIB_QUANT_VALID_END=.*": f"QLIB_QUANT_VALID_END={cfg['valid_end']}",
+    r"QLIB_QUANT_TEST_START=.*": f"QLIB_QUANT_TEST_START={cfg['test_start']}",
+    r"QLIB_QUANT_TEST_END=.*": f"QLIB_QUANT_TEST_END={cfg['test_end']}",
+}
+
+for pattern, replacement in replacements.items():
+    env_text = re.sub(pattern, replacement, env_text)
+
+ENV.write_text(env_text)
+print("✓ .env aktualisiert")
+
+# --- generate.py updaten ---
+data_path = cfg['data_path']
+freq = cfg['frequency']
+train_start = cfg['train_start']
+test_end = cfg['test_end']
+valid_start = cfg['valid_start']
+cols = str(cfg['columns'])
+
+generate_text = f'''import qlib
+import pandas as pd
+
+qlib.init(provider_uri="{data_path}", freq="{freq}")
+
+from qlib.data import D
+
+instruments = D.instruments(market="all")
+fields = {cols}
+
+data = (
+    D.features(instruments, fields, freq="{freq}")
+    .swaplevel()
+    .sort_index()
+    .loc["{train_start}":]
+    .sort_index()
+)
+data.to_hdf("./daily_pv_all.h5", key="data")
+
+data_debug = (
+    D.features(instruments, fields, start_time="{valid_start}", end_time="{test_end}", freq="{freq}")
+    .swaplevel()
+    .sort_index()
+)
+data_debug.to_hdf("./daily_pv_debug.h5", key="data")
+'''
+
+GENERATE.write_text(generate_text)
+print("✓ generate.py aktualisiert")
+print(f"\nKonfiguration angewendet:")
+print(f"  Instrument: {cfg['instrument']}")
+print(f"  Frequenz:   {cfg['frequency']}")
+print(f"  Train:      {cfg['train_start']} → {cfg['train_end']}")
+print(f"  Test:       {cfg['test_start']} → {cfg['test_end']}")

constraints/3.10.txt

@@ -2,7 +2,4 @@ azure-identity==1.17.1
 dill==0.3.9
 pillow==10.4.0
 psutil==6.1.0
-rich==13.9.2
 scipy==1.14.1
-tqdm==4.66.5
-litellm==1.72.4

constraints/3.11.txt

@@ -2,7 +2,4 @@ azure-identity==1.17.1
 dill==0.3.9
 pillow==10.4.0
 psutil==6.1.0
-rich==13.9.2
 scipy==1.14.1
-tqdm==4.66.5
-litellm==1.72.4

data_config.yaml

@@ -0,0 +1,44 @@
+# ============================================================
+#  Predix Data Configuration
+#  Ändere hier Instrument, Frequenz und Zeiträume
+#  Alle anderen Komponenten lesen aus dieser Datei
+# ============================================================
+
+instrument: EURUSD
+frequency: 1min          # 1min, 5min, 15min, 1h, 1d
+data_path: ~/.qlib/qlib_data/eurusd_1min_data
+
+# Verfügbare Spalten (keine $factor Spalte!)
+columns:
+  - $open
+  - $close
+  - $high
+  - $low
+  - $volume
+
+# Walk-Forward Split
+train_start: "2022-03-14"
+train_end:   "2024-06-30"
+valid_start: "2024-07-01"
+valid_end:   "2024-12-31"
+test_start:  "2025-01-01"
+test_end:    "2026-03-20"
+
+# Markt-Kontext für LLM Prompts
+market_context:
+  spread_bps: 1.5
+  sessions:
+    asian:   "00:00-08:00 UTC"
+    london:  "08:00-16:00 UTC"
+    ny:      "13:00-21:00 UTC"
+    overlap: "13:00-16:00 UTC"
+  target_arr: 9.62          # % ARR zu schlagen
+  max_drawdown: 20          # % maximaler Drawdown
+
+# Lookback Referenz (in Bars)
+lookback:
+  1h:  4
+  2h:  8
+  4h:  16
+  8h:  32
+  1d:  96

docs/conf.py

@@ -10,9 +10,9 @@ import subprocess
 
 latest_tag = subprocess.check_output(["git", "describe", "--tags", "--abbrev=0"], text=True).strip()
 
-project = "RDAgent"
-copyright = "2024, Microsoft"
-author = "Microsoft"
+project = "Predix"
+copyright = "2025, Predix Team"
+author = "Predix Team"
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
@@ -66,7 +66,7 @@ html_static_path = ["_static"]
 html_favicon = "_static/favicon.ico"
 
 html_theme_options = {
-    "source_repository": "https://github.com/microsoft/RD-Agent",
+    "source_repository": "https://github.com/PredixAI/predix",
     "source_branch": "main",
     "source_directory": "docs/",
 }

docs/index.rst

@@ -1,13 +1,13 @@
-.. RDAgent documentation master file, created by
+.. Predix documentation master file, created by
    sphinx-quickstart on Mon Jul 15 04:27:50 2024.
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-Welcome to RDAgent's documentation!
+Welcome to Predix's documentation!
 ===================================
 
 .. image:: _static/logo.png
-   :alt: RD-Agent Logo
+   :alt: Predix Logo
 
 .. toctree::
    :maxdepth: 3
@@ -23,7 +23,7 @@ Welcome to RDAgent's documentation!
    api_reference
    policy
 
-   GitHub <https://github.com/microsoft/RD-Agent>
+   GitHub <https://github.com/PredixAI/predix>
 
 
 Indices and tables

docs/installation_and_configuration.rst

@@ -16,6 +16,9 @@ Ensure the current user can run Docker commands **without using sudo**. You can
 LiteLLM Backend Configuration (Default)
 =======================================
 
+.. note::
+   🔥 **Attention**: We now provide experimental support for **DeepSeek** models! You can use DeepSeek's official API for cost-effective and high-performance inference. See the configuration example below for DeepSeek setup.
+
 Option 1: Unified API base for both models
 ------------------------------------------
 
@@ -48,6 +51,23 @@ Option 2: Separate API bases for Chat and Embedding models
       LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
       LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1
 
+Configuration Example: DeepSeek Setup
+-------------------------------------
+
+Many users encounter configuration errors when setting up DeepSeek. Here's a complete working example:
+
+   .. code-block:: Properties
+
+      # CHAT MODEL: Using DeepSeek Official API
+      CHAT_MODEL=deepseek/deepseek-chat 
+      DEEPSEEK_API_KEY=<replace_with_your_deepseek_api_key>
+
+      # EMBEDDING MODEL: Using SiliconFlow for embedding since DeepSeek has no embedding model.
+      # Note: embedding requires litellm_proxy prefix
+      EMBEDDING_MODEL=litellm_proxy/BAAI/bge-m3
+      LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
+      LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1
+
 Necessary parameters include:
 
 - `CHAT_MODEL`: The model name of the chat model.
@@ -87,6 +107,184 @@ Besides, when you are using reasoning models, the response might include the tho
 
 For more details on LiteLLM requirements, refer to the `official LiteLLM documentation <https://docs.litellm.ai/docs>`_.
 
+Configuration Example 2: Azure OpenAI Setup
+-------------------------------------------
+Here’s a sample configuration specifically for Azure OpenAI, based on the `official LiteLLM documentation <https://docs.litellm.ai/docs>`_:
+
+If you're using Azure OpenAI, below is a working example using the Python SDK, following the `LiteLLM Azure OpenAI documentation <https://docs.litellm.ai/docs/providers/azure/>`_:
+
+   .. code-block:: Properties
+
+      from litellm import completion
+      import os
+      
+      # Set Azure OpenAI environment variables
+      os.environ["AZURE_API_KEY"] = "<your_azure_api_key>"
+      os.environ["AZURE_API_BASE"] = "<your_azure_api_base>"
+      os.environ["AZURE_API_VERSION"] = "<version>"
+      
+      # Make a request to your Azure deployment
+      response = completion(
+        "azure/<your_deployment_name>",
+        messages = [{ "content": "Hello, how are you?", "role": "user" }]
+      )
+
+To align with the Python SDK example above, you can configure the `CHAT_MODEL` based on the `response` model setting and use the corresponding `os.environ` variables by writing them into your local `.env` file as follows:
+
+   .. code-block:: Properties
+
+      cat << EOF > .env
+      # CHAT MODEL: Azure OpenAI via LiteLLM
+      CHAT_MODEL=azure/<your_deployment_name>
+      AZURE_API_BASE=https://<your_azure_base>.openai.azure.com/
+      AZURE_API_KEY=<your_azure_api_key>
+      AZURE_API_VERSION=<version>
+      
+      # EMBEDDING MODEL: Using SiliconFlow via litellm_proxy
+      EMBEDDING_MODEL=litellm_proxy/BAAI/bge-large-en-v1.5
+      LITELLM_PROXY_API_KEY=<your_siliconflow_api_key>
+      LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1
+      EOF
+
+This configuration allows you to call Azure OpenAI through LiteLLM while using an external provider (e.g., SiliconFlow) for embeddings.
+
+If your `Azure OpenAI API Key`` supports `embedding model`, you can refer to the following configuration example.
+
+   .. code-block:: Properties
+
+      cat << EOF  > .env
+      EMBEDDING_MODEL=azure/<Model deployment supporting embedding>
+      CHAT_MODEL=azure/<your deployment name>
+      AZURE_API_KEY=<replace_with_your_openai_api_key>
+      AZURE_API_BASE=<your_unified_api_base>
+      AZURE_API_VERSION=<azure api version>
+
+Execution Environment Configuration
+===================================
+
+Coder Environment Configuration (Docker vs. Conda)
+
+RD-Agent's coders can execute code in different environments. You can control this behavior by setting environment variables in your ``.env`` file. This is useful for switching between a local Conda environment and an isolated Docker container.
+
+To configure the environment, add the corresponding line to your ``.env`` file based on the scenario you are running.
+
+**For the Model (Quant) Scenario:**
+
+The execution environment is determined by the ``MODEL_COSTEER_ENV_TYPE`` variable, which is read from ``rdagent/components/coder/model_coder/conf.py``.
+
+*   **To use Docker** (recommended for isolated execution):
+
+    .. code-block:: properties
+
+       MODEL_COSTEER_ENV_TYPE=docker
+
+*   **To use Conda** (for running in a local Conda environment):
+
+    .. code-block:: properties
+
+       MODEL_COSTEER_ENV_TYPE=conda
+
+**For the Data Science Scenario:**
+
+The execution environment is determined by the ``DS_CODER_COSTEER_ENV_TYPE`` variable, which is read from ``rdagent/components/coder/data_science/conf.py``.
+
+*   **To use Docker** (recommended for isolated execution):
+
+    .. code-block:: properties
+
+       DS_CODER_COSTEER_ENV_TYPE=docker
+
+*   **To use Conda** (for running in a local Conda environment):
+
+    .. code-block:: properties
+
+       DS_CODER_COSTEER_ENV_TYPE=conda
+
+
+Custom Time Segment Configuration (Train / Valid / Test)
+=========================================================
+
+RD-Agent now supports user-defined time segments for training, validation,
+and testing (backtesting). Users can customize these segments via environment
+variables in the ``.env`` file, depending on the scenario being executed.
+
+This feature allows greater flexibility when running experiments on different
+time ranges without modifying code or YAML configurations.
+
+Fin-Factor Scenario
+-------------------
+
+When running the **fin_factor** scenario, you can configure the time segments
+using the following environment variables. These variables are read by the
+Factor-related PropSettings and directly affect the execution process.
+
+Add the following entries to your ``.env`` file as needed:
+
+.. code-block:: properties
+
+   QLIB_FACTOR_TRAIN_START=<train start date, default is 2008-01-01>
+   QLIB_FACTOR_TRAIN_END=<train end date, default is 2014-12-31>
+   QLIB_FACTOR_VALID_START=<valid start date, default is 2015-01-01>
+   QLIB_FACTOR_VALID_END=<valid end date, default is 2016-12-31>
+   QLIB_FACTOR_TEST_START=<test / backtest start date, default is 2017-01-01>
+   QLIB_FACTOR_TEST_END=<test / backtest end date, default is 2020-12-31>
+
+Fin-Model Scenario
+------------------
+
+When running the **fin_model** scenario, the model training, validation, and
+testing time segments can be configured independently via the following
+environment variables:
+
+.. code-block:: properties
+
+   QLIB_MODEL_TRAIN_START=<train start date, default is 2008-01-01>
+   QLIB_MODEL_TRAIN_END=<train end date, default is 2014-12-31>
+   QLIB_MODEL_VALID_START=<valid start date, default is 2015-01-01>
+   QLIB_MODEL_VALID_END=<valid end date, default is 2016-12-31>
+   QLIB_MODEL_TEST_START=<test / backtest start date, default is 2017-01-01>
+   QLIB_MODEL_TEST_END=<test / backtest end date, default is 2020-12-31>
+
+These settings are used during model training and evaluation and directly
+impact the execution workflow.
+
+Fin-Quant Scenario
+------------------
+
+When running the **fin_quant** scenario, RD-Agent supports configuring time
+segments for factor, model, and quant stages simultaneously.
+
+**Note:** The ``QLIB_QUANT_*`` variables are only used for front-end UI display
+purposes and do **not** affect the actual execution process.
+
+You may configure the following variables in your ``.env`` file:
+
+.. code-block:: properties
+
+   QLIB_FACTOR_TRAIN_START=<train start date, default is 2008-01-01>
+   QLIB_FACTOR_TRAIN_END=<train end date, default is 2014-12-31>
+   QLIB_FACTOR_VALID_START=<valid start date, default is 2015-01-01>
+   QLIB_FACTOR_VALID_END=<valid end date, default is 2016-12-31>
+   QLIB_FACTOR_TEST_START=<test / backtest start date, default is 2017-01-01>
+   QLIB_FACTOR_TEST_END=<test / backtest end date, default is 2020-12-31>
+
+   QLIB_MODEL_TRAIN_START=<train start date, default is 2008-01-01>
+   QLIB_MODEL_TRAIN_END=<train end date, default is 2014-12-31>
+   QLIB_MODEL_VALID_START=<valid start date, default is 2015-01-01>
+   QLIB_MODEL_VALID_END=<valid end date, default is 2016-12-31>
+   QLIB_MODEL_TEST_START=<test / backtest start date, default is 2017-01-01>
+   QLIB_MODEL_TEST_END=<test / backtest end date, default is 2020-12-31>
+
+   QLIB_QUANT_TRAIN_START=<train start date, default is 2008-01-01>
+   QLIB_QUANT_TRAIN_END=<train end date, default is 2014-12-31>
+   QLIB_QUANT_VALID_START=<valid start date, default is 2015-01-01>
+   QLIB_QUANT_VALID_END=<valid end date, default is 2016-12-31>
+   QLIB_QUANT_TEST_START=<test / backtest start date, default is 2017-01-01>
+   QLIB_QUANT_TEST_END=<test / backtest end date, default is 2020-12-31>
+
+This setup allows the front-end to display consistent segment information
+across different stages while keeping execution logic unchanged.
+
 
 Configuration(deprecated)
 =========================
@@ -95,6 +293,10 @@ To run the application, please create a `.env` file in the root directory of the
 
 If you are using this deprecated version,  you should set `BACKEND` to `rdagent.oai.backend.DeprecBackend`.
 
+   .. code-block:: Properties
+
+      BACKEND=rdagent.oai.backend.DeprecBackend
+
 Here are some other configuration options that you can use:
 
 OpenAI API

docs/scens/catalog.rst

@@ -43,3 +43,4 @@ The supported scenarios are listed below:
     model_agent_fin
     model_copilot_general
     data_science
+    finetune

docs/scens/data_copilot_fin.rst

@@ -125,7 +125,7 @@ You can try our demo by running the following command:
 
     .. code-block:: sh
 
-        rdagent fin_factor_report --report_folder=git_ignore_folder/reports
+        rdagent fin_factor_report --report-folder=git_ignore_folder/reports
 
   - Alternatively, you can store the paths of the reports in `report_result_json_file_path`. The format should be:
 

docs/scens/data_science.rst

@@ -8,12 +8,61 @@ Data Science Agent
 ------------------------------------------------------------------------------------------
 The Data Science Agent is an agent that can automatically perform feature engineering and model tuning. It can be used to solve various data science problems, such as image classification, time series forecasting, and text classification.
 
-🧭 Example Guide
+🌟 Introduction
+~~~~~~~~~~~~~~~~~~
+
+In this scenario, our automated system proposes hypothesis, choose action, implements code, conducts validation, and utilizes feedback in a continuous, iterative process.
+
+The goal is to automatically optimize performance metrics within the validation set or Kaggle Leaderboard, ultimately discovering the most efficient features and models through autonomous research and development.
+
+Here's an enhanced outline of the steps:
+
+**Step 1 : Hypothesis Generation 🔍**
+
+- Generate and propose initial hypotheses based on previous experiment analysis and domain expertise, with thorough reasoning and financial justification.
+
+**Step 2 : Experiment Creation ✨**
+
+- Transform the hypothesis into a task.
+- Choose a specific action within feature engineering or model tuning.
+- Develop, define, and implement a new feature or model, including its name, description, and formulation.
+
+**Step 3 : Model/Feature Implementation 👨‍💻**
+
+- Implement the model code based on the detailed description.
+- Evolve the model iteratively as a developer would, ensuring accuracy and efficiency.
+
+**Step 4 : Validation on Test Set or Kaggle 📉**
+
+- Validate the newly developed model using the test set or Kaggle dataset.
+- Assess the model's effectiveness and performance based on the validation results.
+
+**Step 5: Feedback Analysis 🔍**
+
+- Analyze validation results to assess performance.
+- Use insights to refine hypotheses and enhance the model.
+
+**Step 6: Hypothesis Refinement ♻️**
+
+- Adjust hypotheses based on validation feedback.
+- Iterate the process to continuously improve the model.
+
+📖 Data Science Background
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- 🔧 **Set up RD-Agent Environment**
+In the evolving landscape of artificial intelligence, **Data Science** represents a powerful paradigm where machines engage in autonomous exploration, hypothesis testing, and model development across diverse domains — from healthcare and finance to logistics and research.
 
-  - Before you start, please make sure you have installed RD-Agent and configured the environment for RD-Agent correctly. If you want to know how to install and configure the RD-Agent, please refer to the `documentation <../installation_and_configuration.html>`_.
+The **Data Science** Agent stands as a central engine in this transformation, enabling users to automate the entire machine learning workflow: from hypothesis generation to code implementation, validation, and refinement — all guided by performance feedback.
+
+By leveraging the **Data Science** Agent, researchers and developers can accelerate experimentation cycles. Whether fine-tuning custom models or competing in high-stakes benchmarks like Kaggle, the Data Science Agent unlocks new frontiers in intelligent, self-directed discovery.
+
+🧭 Example Guide - Customized dataset
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+🔧 **Set up RD-Agent Environment**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Before you start, please make sure you have installed RD-Agent and configured the environment for RD-Agent correctly. If you want to know how to install and configure the RD-Agent, please refer to the `documentation <../installation_and_configuration.html>`_.
 
 - 🔩 **Setting the Environment variables at .env file**
 
@@ -24,161 +73,494 @@ The Data Science Agent is an agent that can automatically perform feature engine
     dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
     dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
 
-- 📥 **Prepare Competition Data**
+📥 **Prepare Customized datasets**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-  - Data Science competition data typically consists of three components: a competition description file (in Markdown format), the competition dataset, and evaluation scripts. For reference, an example of a custom user-defined dataset is provided in ``rdagent/scenarios/data_science/example``.
+  - A data science competition dataset usually consists of two parts: ``competition dataset`` and ``evaluation dataset``. (We provide `a sample <https://github.com/microsoft/RD-Agent/tree/main/rdagent/scenarios/data_science/example>`_ of a customized dataset named: `arf-12-hours-prediction-task as a reference`.)
+    
+    - The ``competition dataset`` contains **training data**, **test data**, **description files**, **formatted submission files**, **data sampling codes**.
+    
+    - The ``evaluation dataset`` contains **standard answer file**, **data checking codes**, and **Code for calculation of scores**.
 
-    - **Correct directory structure (Here is an example of competition data with id custom_data)**
+  - We use the ``arf-12-hours-prediction-task`` data as a sample to introduce the preparation workflow for the competition dataset.
+  
+    - Create a ``ds_data/source_data/arf-12-hours-prediction-task`` folder, which will be used to store your raw dataset.
+
+      - The raw files for the competition ``arf-12-hours-prediction-task`` have two files: ``ARF_12h.csv`` and ``X.npz``.
+    
+    - Create a ``ds_data/source_data/arf-12-hours-prediction-task/prepare.py`` file that splits your raw data into **training data**, **test data**, **formatted submission file**, and **standard answer file**. (You will need to write a script based on your raw data.)
+      
+      - The following shows the preprocessing code for the raw data of ``arf-12-hours-prediction-task``.
+
+      .. literalinclude:: ../../rdagent/scenarios/data_science/example/source_data/arf-12-hours-prediction-task/prepare.py
+        :language: python
+        :caption: ds_data/source_data/arf-12-hours-prediction-task/prepare.py
+        :linenos:
+
+      - At the end of program execution, the ``ds_data`` folder structure will look like this:
 
       .. code-block:: text
 
         ds_data
-        └── eval
-        | └── custom_data
-        |    └── grade.py
-        |    └── valid.py
-        |    └── test.csv
-        └── custom_data
-          └── train.csv
-          └── test.csv
-          └── sample_submission.csv
-          └── description.md
-          └── sample.py
-        
-      - ``ds_data/custom_data/train.csv:`` Necessary training data in csv or parquet format, or training images.
+        ├── arf-12-hours-prediction-task
+        │   ├── train
+        │   │   ├── ARF_12h.csv
+        │   │   └── X.npz
+        │   ├── test
+        │   │   ├── ARF_12h.csv
+        │   │   └── X.npz
+        │   └── sample_submission.csv
+        ├── eval
+        │   └── arf-12-hours-prediction-task
+        │       └── submission_test.csv
+        └── source_data
+            └── arf-12-hours-prediction-task
+                ├── ARF_12h.csv
+                ├── prepare.py
+                └── X.npz
 
-      - ``ds_data/custom_data/description.md:`` (Optional) Competition description file.
+    - Create a ``ds_data/arf-12-hours-prediction-task/description.md`` file to describe your competition, Objective, dataset, and other information.
 
-      - ``ds_data/custom_data/sample_submission.csv:`` (Optional) Competition sample submission file.
+      - The following shows the description file for ``arf-12-hours-prediction-task``
 
-      - ``ds_data/custom_data/sample.py:`` (Optional) Sample code for generating debug data from the competition dataset. If not provided, R&D-Agent will use its default sampling logic. For details, see the ``create_debug_data`` function in ``rdagent/scenarios/data_science/debug/data.py``.
+      .. literalinclude:: ../../rdagent/scenarios/data_science/example/arf-12-hours-prediction-task/description.md
+        :language: markdown
+        :caption: ds_data/arf-12-hours-prediction-task/description.md
+        :linenos:
 
-      - ``ds_data/eval/custom_data/grade.py:`` (Optional) Competition grade script, in order to calculate the score for the submission.
+    - Create a ``ds_data/arf-12-hours-prediction-task/sample.py`` file to construct the debugging sample data.
 
-      - ``ds_data/eval/custom_data/valid.py:`` (Optional) Competition validation script, in order to check if the submission format is correct.
+      - The following shows the script for constructing the debugging sample data based on the ``arf-12-hours-prediction-task`` dataset implementation.
 
-      - ``ds_data/eval/custom_data/submission_test.csv:`` (Optional) Competition test label file.
+      .. literalinclude:: ../../rdagent/scenarios/data_science/example/arf-12-hours-prediction-task/sample.py
+        :language: markdown
+        :caption: ds_data/arf-12-hours-prediction-task/sample.py
+        :linenos:
 
-- 🔧 **Set up Environment for Custom User-defined Dataset**
+    - Create a ``ds_data/eval/arf-12-hours-prediction-task/valid.py`` file, which is used to check the validity of the submission files to ensure that their formatting is consistent with the reference file.
+
+      - The following shows a script that checks the validity of a submission based on the ``arf-12-hours-prediction-task`` data.
+
+      .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/arf-12-hours-prediction-task/valid.py
+        :language: markdown
+        :caption: ds_data/eval/arf-12-hours-prediction-task/valid.py
+        :linenos:
+
+    - Create a ``ds_data/eval/arf-12-hours-prediction-task/grade.py`` file, which is used to calculate the score based on the submission file and the **standard answer file**, and output the result in JSON format.
+
+      - The following shows a grading script based on the ``arf-12-hours-prediction-task`` data implementation.
+
+      .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/arf-12-hours-prediction-task/grade.py
+        :language: markdown
+        :caption: ds_data/eval/arf-12-hours-prediction-task/grade.py
+        :linenos:
+
+  - At this point, you have created a complete dataset. The correct structure of the dataset should look like this.
+
+    .. code-block:: text
+
+        ds_data
+        ├── arf-12-hours-prediction-task
+        │   ├── train
+        │   │   ├── ARF_12h.csv
+        │   │   └── X.npz
+        │   ├── test
+        │   │   ├── ARF_12h.csv
+        │   │   └── X.npz
+        │   ├── description.md
+        │   ├── sample_submission.csv
+        │   └── sample.py
+        ├── eval
+        │   └── arf-12-hours-prediction-task
+        │       ├── grade.py
+        │       ├── submission_test.csv
+        │       └── valid.py
+        └── source_data
+            └── arf-12-hours-prediction-task
+                ├── ARF_12h.csv
+                ├── prepare.py
+                └── X.npz
+
+  - The above shows the complete dataset creation workflow, some of the files are not required, in practice you can customize the dataset according to your own needs.
+
+    - If we don't need the test set scores, then we can choose not to generate **formatted submission files** and **standard answer file** in the prepare code, and we don't need to write **data checking codes** and **Code for calculation of scores**.
+
+    - **Data sampling code** can also be created according to the actual need, if you do not provide **data sampling code**, RD-Agent will be handed over to the LLM sampling at runtime.
+
+      - In the default sampling method (``create_debug_data``), the default sampling ratio (parameter: ``min_frac``) is 1%, if 1% of the data is less than 5, then 5 data will be sampled (parameter: ``min_num``), you can adjust the sampling ratio by adjusting these two parameters.
+
+        - If you have customized data sampling code, you need to set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` (default is True) in the ``.env`` file before running, so that the program will use the customized sampling code when running, and you can just execute this line of code in the command line:
+
+          .. code-block:: sh
+
+            dotenv set DS_SAMPLE_DATA_BY_LLM False
+
+        - In addition, we provide a data sampling method in `rdagent.scenarios.data_science.debug.data.create_debug_data <https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/data_science/debug/data.py#L605>`_, in this method, the default sampling ratio (parameter: ``min_frac``) is 1%, if 1% of the data is less than 5, then 5 data will be sampled (parameter: ``min_num``), you can use this method by the following two ways.
+
+          - You can set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` in the ``.env`` file so that when the program runs, it will use the sampling code provided by RD-Agent.
+
+            .. code-block:: sh
+
+              dotenv set DS_SAMPLE_DATA_BY_LLM False
+
+          - If you think that the parameters in the receipt sampling method provided by RD-Agent are not suitable, you can customize the parameters in the following command and run it, and set ``DS_SAMPLE_DATA_BY_LLM`` to ``False`` in the ``.env`` so that the program will use the sampling data you provided when running.
+
+            .. code-block:: sh
+
+              python rdagent/app/data_science/debug.py --dataset_path <dataset path> --competition <competiton_name> --min_frac <sampling ratio> --min_num <minimum number of sampling>
+              dotenv set DS_SAMPLE_DATA_BY_LLM False
+
+  - If you don't need the scores from the test set and leave the data sampling to the LLM, or if you use the sampling method provided by the RD-Agent, you only need to prepare a minimal dataset. The structure of the simplest dataset should be as shown below.
+
+    .. code-block:: text
+
+        ds_data
+        ├── arf-12-hours-prediction-task
+        │   ├── train
+        │   │   ├── ARF_12h.csv
+        │   │   └── X.npz
+        │   ├── test
+        │   │   ├── ARF_12h.csv
+        │   │   └── X.npz
+        │   └── description.md
+        └── source_data
+            └── arf-12-hours-prediction-task
+                ├── ARF_12h.csv
+                ├── prepare.py
+                └── X.npz
+
+  - We have prepared a dataset based on the above description for your reference. You can download it with the following command.
+
+    .. code-block:: sh
+
+      wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip
+
+⚙️ **Set up Environment for Customized datasets**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
   .. code-block:: sh
 
       dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen
-      dotenv set DS_LOCAL_DATA_PATH rdagent/scenarios/data_science/example
-      dotenv set DS_IF_USING_MLE_DATA False
+      dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
       dotenv set DS_CODER_ON_WHOLE_PIPELINE True
-      dotenv set DS_CODER_COSTEER_ENV_TYPE docker
 
-- 🚀 **Run the Application**
+  - 📘 More Environment Variables (Optional)
 
-  - You can directly run the application by using the following command:
+    - If you want to see all the available environment variables, you can refer to the configuration file for Data Science scenarios:
+
+    .. literalinclude:: ../../rdagent/app/data_science/conf.py
+      :language: python
+      :linenos:
+
+    - These variables allow you to have finer-grained control in Data Science scenarios.
+
+🚀 **Run the Application**
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  - 🌏 You can directly run the application by using the following command:
     
     .. code-block:: sh
 
         rdagent data_science --competition <Competition ID>
 
-  - Then, you can run the test set score corresponding to each round of the loop.
-
-    .. code-block:: sh
-
-        dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>
-
-    Here, <url_to_log> refers to the parent directory of the log folder generated during the run.
-
-- 📥 **Visualize the R&D Process**
-
-  - We provide a web UI to visualize the log. You just need to run:
-
-    .. code-block:: sh
-
-        streamlit run rdagent/log/ui/dsapp.py
-
-  - Then you can input the log path and visualize the R&D process.
-
-🔍 MLE-bench Guide: Running ML Engineering via MLE-bench
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- 📝 **MLE-bench Overview**
-
-  - MLE-bench is a comprehensive benchmark designed to evaluate the ML engineering capabilities of AI systems using real-world scenarios. The dataset comprises 75 Kaggle competitions. Since Kaggle does not provide held-out test sets for these competitions, the benchmark includes preparation scripts that split the publicly available training data into new training and test sets, and grading scripts are provided for each competition to accurately evaluate submission scores.
-
-- 🔧 **Set up Environment for MLE-bench**
-
-  - Running R&D-Agent on MLE-bench is designed for full automation. There is no need for manual downloads and data preparation. Simply set the environment variable ``DS_IF_USING_MLE_DATA`` to True.  
-
-  - At runtime, R&D-Agent will automatically build the Docker image specified at ``rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile``. This image is responsible for downloading the required datasets and grading files for MLE-bench.  
-  
-  - Note: The first run may take longer than subsequent runs as the Docker image and data are being downloaded and set up for the first time.
-
-    .. code-block:: sh
-
-        dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
-        dotenv set DS_IF_USING_MLE_DATA True
-
-- 🔨 **Configuring the Kaggle API**
-
-  - Downloading Kaggle competition data requires the Kaggle API. You can set up the Kaggle API by following these steps:
-  
-    - Register and login on the `Kaggle <https://www.kaggle.com/>`_ website.
-
-    - Click on the avatar (usually in the top right corner of the page) -> ``Settings`` -> ``Create New Token``, A file called ``kaggle.json`` will be downloaded.
-
-    - Move ``kaggle.json`` to ``~/.config/kaggle/``
-
-    - Modify the permissions of the ``kaggle.json`` file.
+    - The following shows the command to run based on the ``arf-12-hours-prediction-task`` data
 
       .. code-block:: sh
 
-        chmod 600 ~/.config/kaggle/kaggle.json
+          rdagent data_science --competition arf-12-hours-prediction-task
 
-  - For more information about Kaggle API Settings, refer to the `Kaggle API <https://github.com/Kaggle/kaggle-api>`_.
+    -  More CLI Parameters for `rdagent data_science` command:
 
+      .. automodule:: rdagent.app.data_science.loop
+        :members:
+        :no-index:
 
-- 🔩 **Setting the Environment Variables for MLE-bench**
+  - 📈 Visualize the R&D Process
 
-  - In addition to auto-downloading the benchmark data, you must also configure the runtime environment for executing the competition code.  
-  - Use the environment variable ``DS_CODER_COSTEER_ENV_TYPE`` to select the execution mode:
-    
-    • When set to docker (the default), RD-Agent utilizes the official Kaggle Docker image (``gcr.io/kaggle-gpu-images/python:latest``) to ensure that all required packages are available.  
-    • If you prefer to use a custom Docker setup, you can modify the configuration using ``DS_DOCKER_IMAGE`` or ``DS_DOCKERFILE_FOLDER_PATH``.  
-    • Alternatively, if your competition work only demands basic libraries, you may set ``DS_CODER_COSTEER_ENV_TYPE`` to conda. In this mode, you must create a local conda environment named “kaggle” and pre-install the necessary packages. RD-Agent will execute the competition code within this “kaggle” conda environment.
+    - We provide a web UI to visualize the log. You just need to run:
+
+      .. code-block:: sh
+
+          rdagent ui --port <custom port> --log-dir <your log folder like "log/"> --data_science True
+
+    - Then you can input the log path and visualize the R&D process.
+
+  - 🧪 Scoring the test results
+
+    - Finally, shutdown the program, and get the test set scores with this command.
 
     .. code-block:: sh
 
-      # Configure the runtime environment: choice between 'docker' (default) or 'conda'
-      dotenv set DS_CODER_COSTEER_ENV_TYPE docker
+      dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>
 
-- 🚀 **Run the Application**
+    Here, <url_to_log> refers to the parent directory of the log folder generated during the run.
+
+🕹️ Kaggle Agent
+~~~~~~~~~~~~~~~~
+
+📖 Background
+^^^^^^^^^^^^^^
+
+In the landscape of data science competitions, Kaggle serves as the ultimate arena where data enthusiasts harness the power of algorithms to tackle real-world challenges.
+The Kaggle Agent stands as a pivotal tool, empowering participants to seamlessly integrate cutting-edge models and datasets, transforming raw data into actionable insights.
+
+By utilizing the **Kaggle Agent**, data scientists can craft innovative solutions that not only uncover hidden patterns but also drive significant advancements in predictive accuracy and model robustness.
+
+🧭 Example Guide - Kaggle Dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+🛠️ Preparing For The Competition
+""""""""""""""""""""""""""""""""""
+
+- 🔨 **Configuring the Kaggle API**
+
+  - Register and login on the `Kaggle <https://www.kaggle.com/>`_ website.
+  - Click on the avatar (usually in the top right corner of the page) -> ``Settings`` -> ``Create New Token``, A file called ``kaggle.json`` will be downloaded.
+  - Move ``kaggle.json`` to ``~/.config/kaggle/``
+  - Modify the permissions of the ``kaggle.json`` file.
+
+    .. code-block:: sh
+
+      chmod 600 ~/.config/kaggle/kaggle.json
+
+  - For more information about Kaggle API Settings, refer to the `Kaggle API <https://github.com/Kaggle/kaggle-api>`_.
+
+- 🔩 **Setting the Environment variables at .env file**
+
+  - Determine the path where the data will be stored and add it to the ``.env`` file.
+
+  .. code-block:: sh
+
+    mkdir -p <your local directory>/ds_data
+    dotenv set KG_LOCAL_DATA_PATH <your local directory>/ds_data
+
+  - 📘 More Environment Variables (Optional)
+
+    - If you want to see all the available environment variables, you can refer to the configuration file for Data Science scenarios:
+
+    .. literalinclude:: ../../rdagent/app/data_science/conf.py
+      :language: python
+      :linenos:
+
+    - These variables allow you to have finer-grained control in Data Science scenarios.
+
+- 🗳️ **Join the competition**
+
+  - If your Kaggle API account has not joined a competition, you will need to join the competition before running the program.
+
+    - At the bottom of the competition details page, you can find the ``Join the competition`` button, click on it and select ``I Understand and Accept`` to join the competition.
+
+    - In the **Competition List Available** below, you can jump to the competition details page.
+
+📥 Preparing Competition DataDataset && Set up RD-Agent Environment
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+- As a subset of data science, kaggle's dataset still follows the data science format. Based on this, the kaggle dataset can be divided into two categories depending on whether or not it is supported by the **MLE-Bench**.
+
+  - What is **MLE-Bench**?
+
+    - **MLE-Bench** is a comprehensive benchmark designed to evaluate the **machine learning engineering** capabilities of AI systems using real-world scenarios. The dataset includes multiple Kaggle competitions. Since Kaggle does not provide reserved test sets for these competitions, the benchmark includes preparation scripts for splitting publicly available training data into new training and test sets, and scoring scripts for each competition to accurately evaluate submission scores.
+
+  - I'm running a competition Is **MLE-Bench** supported?
+
+    - You can see all the competitions supported by **MLE-Bench** `here <https://github.com/openai/mle-bench/tree/main/mlebench/competitions>`_.
+
+- Prepare datasets for **MLE-Bench** supported competitions.
+
+  - If you agree with the **MLE-Bench** standard, then you don't need to prepare the dataset, you just need to configure your ``.env`` file to automate the download of the dataset.
+
+    - Configure environment variables, add ``DS_IF_USING_MLE_DATA`` to environment variables, and set it to ``True``.
+
+      .. code-block:: sh
+
+        dotenv set DS_IF_USING_MLE_DATA True
+
+    - Configure environment variables, add ``DS_SAMPLE_DATA_BY_LLM`` to environment variables, and set it to ``True``.
+
+      .. code-block:: sh
+
+        dotenv set DS_SAMPLE_DATA_BY_LLM True
+
+    - Configure environment variables, add ``DS_SCEN`` to environment variables, and set it to ``rdagent.scenarios.data_science.scen.KaggleScen``.
+
+      .. code-block:: sh
+
+        dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen
+
+  - At this point, you are ready to start running your competition, which will automatically download the data, and the LLM will automatically extract the minimum dataset.
+
+    - After running the program the structure of the ds_data folder should look like this (Using the ``tabular-playground-series-dec-2021`` contest as an example).
+
+      .. code-block:: text
+
+        ds_data
+        ├── tabular-playground-series-dec-2021
+        │   ├── description.md
+        │   ├── sample_submission.csv
+        │   ├── test.csv
+        │   └── train.csv
+        └── zip_files
+            └── tabular-playground-series-dec-2021
+                └── tabular-playground-series-dec-2021.zip
+
+      - The ``ds_data/zip_files`` folder contains a zip file of the raw competition data downloaded from kaggle website.
+
+  - At runtime, RD-Agent will automatically build the Docker image specified at `rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile <https://github.com/microsoft/RD-Agent/blob/main/rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile>`_. This image is responsible for downloading the required datasets and grading files for MLE-Bench.
+
+  Note: The first run may take longer than subsequent runs as the Docker image and data are being downloaded and set up for the first time.
+
+- Prepare datasets for competitions that are not supported by **MLE-Bench**.
+
+  - As a subset of data science, we can follow the format and steps of data science dataset to prepare kaggle dataset. Below we will describe the workflow for preparing a kaggle dataset using the competition ``playground-series-s4e9`` as an example.
+  
+    - Create a ``ds_data/source_data/playground-series-s4e9`` folder, which will be used to store your raw dataset.
+
+      - The raw files for the competition ``playground-series-s4e9`` have two files: ``train.csv``, ``test.csv``, ``sample_submission.csv``, and there are two ways to get the raw data:
+
+        - You can find the raw data required for the competition on the `official kaggle website <https://www.kaggle.com/competitions/playground-series-s4e9/data>`_.
+
+        - Or you can use the command line to download the raw data for the competition, the download command is as follows.
+
+          .. code-block:: sh
+
+            kaggle competitions download -c playground-series-s4e9
+
+    - Create a ``ds_data/source_data/playground-series-s4e9/prepare.py`` file that splits your raw data into **training data**, **test data**, **formatted submission file**, and **standard answer file**. (You will need to write a script based on your raw data.)
+
+      - The following shows the preprocessing code for the raw data of ``playground-series-s4e9``.
+
+      .. literalinclude:: ../../rdagent/scenarios/data_science/example/source_data/playground-series-s4e9/prepare.py
+        :language: python
+        :caption: ds_data/source_data/playground-series-s4e9/prepare.py
+        :linenos:
+
+      - At the end of program execution, the ``ds_data`` folder structure will look like this:
+
+      .. code-block:: text
+
+        ds_data
+        ├── playground-series-s4e9
+        │   ├── train.csv
+        │   ├── test.csv
+        │   └── sample_submission.csv
+        ├── eval
+        │   └── playground-series-s4e9
+        │       └── submission_test.csv
+        └── source_data
+            └── playground-series-s4e9
+                ├── prepare.py
+                ├── sample_submission.csv
+                ├── test.csv
+                └── train.csv
+
+    - Create a ``ds_data/playground-series-s4e9/description.md`` file to describe your competition, dataset description, and other information. We can find the `competition description information <https://www.kaggle.com/competitions/playground-series-s4e9/overview>`_ and the `dataset description information <https://www.kaggle.com/competitions/playground-series-s4e9/data>`_ from the Kaggle website.
+
+      - The following shows the description file for ``playground-series-s4e9``
+
+        .. literalinclude:: ../../rdagent/scenarios/data_science/example/playground-series-s4e9/description.md
+          :language: markdown
+          :caption: ds_data/playground-series-s4e9/description.md
+          :linenos:
+
+    - Create a ``ds_data/eval/playground-series-s4e9/valid.py`` file, which is used to check the validity of the submission files to ensure that their formatting is consistent with the reference file.
+
+      - The following shows a script that checks the validity of a submission based on the ``playground-series-s4e9`` data.
+
+      .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/playground-series-s4e9/valid.py
+        :language: markdown
+        :caption: ds_data/eval/playground-series-s4e9/valid.py
+        :linenos:
+
+    - Create a ``ds_data/eval/playground-series-s4e9/grade.py`` file, which is used to calculate the score based on the submission file and the **standard answer file**, and output the result in JSON format.
+
+      - The following shows a grading script based on the ``playground-series-s4e9`` data implementation.
+
+      .. literalinclude:: ../../rdagent/scenarios/data_science/example/eval/playground-series-s4e9/grade.py
+        :language: markdown
+        :caption: ds_data/eval/playground-series-s4e9/grade.py
+        :linenos:
+
+  - In this example we don't create a ``ds_data/eval/playground-series-s4e9/sample.py``, we use the sample method provided by RD-Agent by default.
+
+  - At this point, you have created a complete dataset. The correct structure of the dataset should look like this.
+
+    .. code-block:: text
+
+        ds_data
+        ├── playground-series-s4e9
+        │   ├── train.csv
+        │   ├── test.csv
+        │   ├── description.md
+        │   └── sample_submission.csv
+        ├── eval
+        │   └── playground-series-s4e9
+        │       ├── grade.py
+        │       ├── submission_test.csv
+        │       └── valid.py
+        └── source_data
+            └── playground-series-s4e9
+                ├── prepare.py
+                ├── sample_submission.csv
+                ├── test.csv
+                └── train.csv
+
+  - We have prepared a dataset based on the above description for your reference. You can download it with the following command.
+
+    .. code-block:: sh
+
+      wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/playground-series-s4e9.zip
+
+  - Next, we need to configure the environment for the ``playground-series-s4e9`` contest. You can do this by executing the following command at the command line.
+
+    .. code-block:: sh
+
+      dotenv set DS_IF_USING_MLE_DATA False
+      dotenv set DS_SAMPLE_DATA_BY_LLM False
+      dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen
+
+🚀 **Run the Application**
+""""""""""""""""""""""""""""""""""""
+
+  - 🌏 You can directly run the application by using the following command:
 
-  - You can directly run the application by using the following command:
-    
     .. code-block:: sh
 
         rdagent data_science --competition <Competition ID>
 
-- 📥 **Visualize the R&D Process**
+    - The following shows the command to run based on the ``playground-series-s4e9`` data
 
-  - We provide a web UI to visualize the log. You just need to run:
+      .. code-block:: sh
+
+          rdagent data_science --competition playground-series-s4e9
+
+    -  More CLI Parameters for `rdagent data_science` command:
+
+      .. automodule:: rdagent.app.data_science.loop
+        :members:
+        :no-index:
+
+  - 📈 Visualize the R&D Process
+
+    - We provide a web UI to visualize the log. You just need to run:
+
+      .. code-block:: sh
+
+          rdagent ui --port <custom port> --log-dir <your log folder like "log/"> --data_science True
+
+    - Then you can input the log path and visualize the R&D process.
+
+  - 🧪 Scoring the test results
+
+    - Finally, shutdown the program, and get the test set scores with this command.
 
     .. code-block:: sh
 
-        streamlit run rdagent/log/ui/dsapp.py
+      dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>
 
-  - Then you can input the log path and visualize the R&D process.
+    - If you have configured the full output in ``ds_data/eval/playground-series-s4e9/grade.py``, or if you are running a competition that receives **MLE-Bench** support, you can also summarize the scores by running the following command.
 
-- **Additional Guidance**
-
-  - **Combine different LLM Models at R&D Stage**
-
-    - You can combine different LLM models at the R&D stage. 
-
-    - By default, when you set environment variable ``CHAT_MODEL``, it covers both R&D stages. When customizing the model for the development stage, you can set:
-    
     .. code-block:: sh
 
-      # This example sets the model to "o3-mini". For some models, the reasoning effort shoule be set to "None".
-      dotenv set LITELLM_CHAT_MODEL_MAP '{"coding":{"model":"o3-mini","reasoning_effort":"high"},"running":{"model":"o3-mini","reasoning_effort":"high"}}'
-
-
-
+      rdagent grade_summary --log-folder=<url_to_log>
 
+    Here, <url_to_log> refers to the parent directory of the log folder generated during the run.

docs/scens/finetune.rst

@@ -0,0 +1,163 @@
+.. _finetune_agent:
+
+=============================
+Fine-tuning an Existing Model
+=============================
+
+## **🎯 Scenario: Continue Training on a Pre-trained Model**
+
+In this workflow the **Data Science Agent** starts from a *previously trained* model (and its training script), performs additional fine-tuning on new data, and then re-uses the updated weights for subsequent inference runs.
+
+🚧 Directory Structure
+
+Your competition folder (here called ``custom_data``) must contain **one extra sub-directory** named ``prev_model`` where you keep the old weights and the code that produced them:
+
+.. code-block:: text
+
+   ds_data
+   └── custom_data
+       ├── train.csv
+       ├── test.csv
+       ├── sample_submission.csv      # optional
+       ├── description.md             # optional
+       ├── sample.py                  # optional
+       └── prev_model                 # ← NEW
+           ├── models/                #   previous checkpoints (e.g. *.bin, *.pt, *.ckpt)
+           └── main.py                  #   training/inference scripts you used before
+
+If your competition provides custom grading/validation scripts, keep them under ``ds_data/eval/custom_data`` exactly as before.
+
+🔧 Environment Setup
+~~~~~~~~~~~~~~~~~~~~~~
+
+Add or update the following variables in **.env** (examples shown):
+
+.. code-block:: sh
+
+   # required for all Data-Science runs
+   dotenv set DS_LOCAL_DATA_PATH <your local path>/ds_data
+
+   # optional: choose docker / conda, etc.
+   dotenv set DS_CODER_COSTEER_ENV_TYPE docker
+
+🚀 How It Works at Runtime
+
+1. **First run**
+
+   * `rdagent` detects `prev_model/models`.
+   * It loads the latest checkpoint and prepare the fine-tuning based on code found under `prev_model/*.py` (or your own pipeline if you override it).
+   * Fine-tuned weights are written to `./workspace_input/models`.
+
+2. **Subsequent runs**
+
+   * When you execute `python ./workspace_input/main.py`, the script first looks for a checkpoint in `./workspace_input/models`.
+   * If found, it **skips fine-tuning** and goes straight to prediction / submission generation.
+
+⏰ Managing Timeouts
+
+
+By default:
+
+* **Debug loop**: 1 hour (``DS_DEBUG_TIMEOUT=3600`` seconds)  
+* **Full run**  : 3 hours (``DS_FULL_TIMEOUT=10800`` seconds)
+
+Override either value in **.env**:
+
+.. code-block:: sh
+
+   # give the debug loop 45 min and the full loop 6 h
+   dotenv set DS_DEBUG_TIMEOUT 2700
+   dotenv set DS_FULL_TIMEOUT 21600
+
+- 🚀 **Run the Application**
+
+  - You can directly run the application by using the following command:
+    
+    .. code-block:: sh
+
+        dotenv run -- python rdagent/app/finetune/data_science/loop.py --competition <Competition ID>
+
+  - Then, you can run the test set score corresponding to each round of the loop.
+
+    .. code-block:: sh
+
+        dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>
+
+    Here, <url_to_log> refers to the parent directory of the log folder generated during the run.
+
+- 📥 **Visualize the R&D Process**
+
+  - We provide a web UI to visualize the log. You just need to run:
+
+    .. code-block:: sh
+
+        streamlit run rdagent/log/ui/dsapp.py
+
+  - Then you can input the log path and visualize the R&D process.
+
+🔍 MLE-bench Guide: Running ML Engineering via MLE-bench
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- 📝 **MLE-bench Overview**
+
+  - MLE-bench is a comprehensive benchmark designed to evaluate the ML engineering capabilities of AI systems using real-world scenarios. The dataset comprises 75 Kaggle competitions. Since Kaggle does not provide held-out test sets for these competitions, the benchmark includes preparation scripts that split the publicly available training data into new training and test sets, and grading scripts are provided for each competition to accurately evaluate submission scores.
+
+- 🔧 **Set up Environment for MLE-bench**
+
+  - Running R&D-Agent on MLE-bench is designed for full automation. There is no need for manual downloads and data preparation. Simply set the environment variable ``DS_IF_USING_MLE_DATA`` to True.  
+
+  - At runtime, R&D-Agent will automatically build the Docker image specified at ``rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile``. This image is responsible for downloading the required datasets and grading files for MLE-bench.  
+  
+  - Note: The first run may take longer than subsequent runs as the Docker image and data are being downloaded and set up for the first time.
+
+    .. code-block:: sh
+
+        dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
+        dotenv set DS_IF_USING_MLE_DATA True
+
+- 🔨 **Configuring the Kaggle API**
+
+  - Downloading Kaggle competition data requires the Kaggle API. You can set up the Kaggle API by following these steps:
+  
+    - Register and login on the `Kaggle <https://www.kaggle.com/>`_ website.
+
+    - Click on the avatar (usually in the top right corner of the page) -> ``Settings`` -> ``Create New Token``, A file called ``kaggle.json`` will be downloaded.
+
+    - Move ``kaggle.json`` to ``~/.config/kaggle/``
+
+    - Modify the permissions of the ``kaggle.json`` file.
+
+      .. code-block:: sh
+
+        chmod 600 ~/.config/kaggle/kaggle.json
+
+  - For more information about Kaggle API Settings, refer to the `Kaggle API <https://github.com/Kaggle/kaggle-api>`_.
+
+
+- 🔩 **Setting the Environment Variables for MLE-bench**
+
+  - In addition to auto-downloading the benchmark data, you must also configure the runtime environment for executing the competition code.  
+  - Use the environment variable ``DS_CODER_COSTEER_ENV_TYPE`` to select the execution mode:
+    
+    • When set to docker (the default), RD-Agent utilizes the official Kaggle Docker image (``gcr.io/kaggle-gpu-images/python:latest``) to ensure that all required packages are available.  
+    • If you prefer to use a custom Docker setup, you can modify the configuration using ``DS_DOCKER_IMAGE`` or ``DS_DOCKERFILE_FOLDER_PATH``.  
+    • Alternatively, if your competition work only demands basic libraries, you may set ``DS_CODER_COSTEER_ENV_TYPE`` to conda. In this mode, you must create a local conda environment named “kaggle” and pre-install the necessary packages. RD-Agent will execute the competition code within this “kaggle” conda environment.
+
+    .. code-block:: sh
+
+      # Configure the runtime environment: choice between 'docker' (default) or 'conda'
+      dotenv set DS_CODER_COSTEER_ENV_TYPE docker
+
+- **Additional Guidance**
+
+  - **Combine different LLM Models at R&D Stage**
+
+    - You can combine different LLM models at the R&D stage. 
+
+    - By default, when you set environment variable ``CHAT_MODEL``, it covers both R&D stages. When customizing the model for the development stage, you can set:
+    
+    .. code-block:: sh
+
+      # This example sets the model to "o3-mini". For some models, the reasoning effort shoule be set to "None".
+      dotenv set LITELLM_CHAT_MODEL_MAP '{"coding":{"model":"o3-mini","reasoning_effort":"high"},"running":{"model":"o3-mini","reasoning_effort":"high"}}'
+

docs/scens/model_copilot_general.rst

@@ -96,4 +96,4 @@ You can try our demo by running the following command:
   
     .. code-block:: sh
 
-        rdagent general_model --report_file_path=<path_to_pdf_file>
+        rdagent general_model --report-file-path=<path_to_pdf_file>

docs/ui.rst

@@ -18,7 +18,7 @@ In `RD-Agent/` folder, run:
 
 .. code-block:: bash
 
-    rdagent ui --port <port> --log_dir <log_dir like "log/"> [--debug]
+    rdagent ui --port <port> --log-dir <log_dir like "log/"> [--debug]
 
 This will start a web app on `http://localhost:<port>`.
 

patches/eva_utils.py

@@ -0,0 +1,553 @@
+import io
+import json
+from abc import abstractmethod
+from typing import Dict, Tuple
+
+import pandas as pd
+
+from rdagent.components.coder.factor_coder.config import FACTOR_COSTEER_SETTINGS
+from rdagent.components.coder.factor_coder.factor import FactorTask
+from rdagent.core.experiment import Task, Workspace
+from rdagent.oai.llm_conf import LLM_SETTINGS
+from rdagent.oai.llm_utils import APIBackend
+from rdagent.utils.agent.tpl import T
+
+
+class FactorEvaluator:
+    """Although the init method is same to Evaluator, but we want to emphasize they are different"""
+
+    def __init__(self, scen=None) -> None:
+        self.scen = scen
+
+    @abstractmethod
+    def evaluate(
+        self,
+        target_task: Task,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+        **kwargs,
+    ) -> Tuple[str, object]:
+        """You can get the dataframe by
+
+        .. code-block:: python
+
+            _, gen_df = implementation.execute()
+            _, gt_df = gt_implementation.execute()
+
+        Returns
+        -------
+        Tuple[str, object]
+            - str: the text-based description of the evaluation result
+            - object: a comparable metric (bool, integer, float ...) None for evaluator with only text-based result
+
+        """
+        raise NotImplementedError("Please implement the `evaluator` method")
+
+    def _get_df(self, gt_implementation: Workspace, implementation: Workspace):
+        if gt_implementation is not None:
+            _, gt_df = gt_implementation.execute()
+            if isinstance(gt_df, pd.Series):
+                gt_df = gt_df.to_frame("gt_factor")
+            if isinstance(gt_df, pd.DataFrame):
+                gt_df = gt_df.sort_index()
+        else:
+            gt_df = None
+
+        _, gen_df = implementation.execute()
+        if isinstance(gen_df, pd.Series):
+            gen_df = gen_df.to_frame("source_factor")
+        if isinstance(gen_df, pd.DataFrame):
+            gen_df = gen_df.sort_index()
+        return gt_df, gen_df
+
+    def __str__(self) -> str:
+        return self.__class__.__name__
+
+
+class FactorCodeEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        target_task: FactorTask,
+        implementation: Workspace,
+        execution_feedback: str,
+        value_feedback: str = "",
+        gt_implementation: Workspace = None,
+        **kwargs,
+    ):
+        factor_information = target_task.get_task_information()
+        code = implementation.all_codes
+
+        system_prompt = T(".prompts:evaluator_code_feedback_v1_system").r(
+            scenario=(
+                self.scen.get_scenario_all_desc(
+                    target_task,
+                    filtered_tag="feature",
+                    simple_background=FACTOR_COSTEER_SETTINGS.simple_background,
+                )
+                if self.scen is not None
+                else "No scenario description."
+            )
+        )
+
+        execution_feedback_to_render = execution_feedback
+        for _ in range(10):  # 10 times to split the content is enough
+            user_prompt = T(".prompts:evaluator_code_feedback_v1_user").r(
+                factor_information=factor_information,
+                code=code,
+                execution_feedback=execution_feedback_to_render,
+                value_feedback=value_feedback,
+                gt_code=gt_implementation.code if gt_implementation else None,
+            )
+            if (
+                APIBackend().build_messages_and_calculate_token(
+                    user_prompt=user_prompt,
+                    system_prompt=system_prompt,
+                )
+                > APIBackend().chat_token_limit
+            ):
+                execution_feedback_to_render = execution_feedback_to_render[len(execution_feedback_to_render) // 2 :]
+            else:
+                break
+        critic_response = APIBackend().build_messages_and_create_chat_completion(
+            user_prompt=user_prompt,
+            system_prompt=system_prompt,
+            json_mode=False,
+        )
+
+        return critic_response, None
+
+
+class FactorInfEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str, object]:
+        _, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return (
+                "The source dataframe is None. Please check the implementation.",
+                False,
+            )
+        INF_count = gen_df.isin([float("inf"), -float("inf")]).sum().sum()
+        if INF_count == 0:
+            return "The source dataframe does not have any infinite values.", True
+        else:
+            return (
+                f"The source dataframe has {INF_count} infinite values. Please check the implementation.",
+                False,
+            )
+
+
+class FactorSingleColumnEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str, object]:
+        _, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return (
+                "The source dataframe is None. Please check the implementation.",
+                False,
+            )
+        if len(gen_df.columns) == 1:
+            return "The source dataframe has only one column which is correct.", True
+        else:
+            return (
+                "The source dataframe has more than one column. Please check the implementation. We only evaluate the first column.",
+                False,
+            )
+
+
+class FactorOutputFormatEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str, object]:
+        gt_df, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return (
+                "The source dataframe is None. Skip the evaluation of the output format.",
+                False,
+            )
+        buffer = io.StringIO()
+        gen_df.info(buf=buffer)
+        gen_df_info_str = f"The user is currently working on a feature related task.\nThe output dataframe info is:\n{buffer.getvalue()}"
+        system_prompt = T(".prompts:evaluator_output_format_system").r(
+            scenario=(
+                self.scen.get_scenario_all_desc(implementation.target_task, filtered_tag="feature")
+                if self.scen is not None
+                else "No scenario description."
+            )
+        )
+
+        # TODO: with retry_context(retry_n=3, except_list=[KeyError]):
+        max_attempts = 3
+        attempts = 0
+        final_evaluation_dict = None
+
+        while attempts < max_attempts:
+            try:
+                api = APIBackend() if attempts == 0 else APIBackend(use_chat_cache=False)
+                resp = api.build_messages_and_create_chat_completion(
+                    user_prompt=gen_df_info_str,
+                    system_prompt=system_prompt,
+                    json_mode=True,
+                    json_target_type=Dict[str, str | bool | int],
+                )
+                resp_dict = json.loads(resp)
+                resp_dict["output_format_decision"] = str(resp_dict["output_format_decision"]).lower() in ["true", "1"]
+
+                return (
+                    str(resp_dict["output_format_feedback"]),
+                    resp_dict["output_format_decision"],
+                )
+            except (KeyError, json.JSONDecodeError) as e:
+                attempts += 1
+                if attempts >= max_attempts:
+                    raise KeyError(
+                        "Wrong JSON Response or missing 'output_format_decision' or 'output_format_feedback' key after multiple attempts."
+                    ) from e
+
+        return "Failed to evaluate output format after multiple attempts.", False
+
+
+class FactorDatetimeDailyEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str | object]:
+        _, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return "The source dataframe is None. Skip the evaluation of the datetime format.", False
+
+        if "datetime" not in gen_df.index.names:
+            return "The source dataframe does not have a datetime index. Please check the implementation.", False
+
+        try:
+            pd.to_datetime(gen_df.index.get_level_values("datetime"))
+        except Exception:
+            return (
+                f"The source dataframe has a datetime index but it is not in the correct format (maybe a regular string or other objects). Please check the implementation.\n The head of the output dataframe is: \n{gen_df.head()}",
+                False,
+            )
+
+        time_diff = pd.to_datetime(gen_df.index.get_level_values("datetime")).to_series().diff().dropna()
+        min_diff = time_diff.min()
+        if min_diff <= pd.Timedelta(minutes=1):
+            return (
+                "The generated dataframe is not daily. The implementation is definitely wrong. Please check the implementation.",
+                False,
+            )
+        if min_diff <= pd.Timedelta(minutes=30):
+            return "The generated dataframe is intraday (1min bars). This is correct for EURUSD.", True
+        return "The generated dataframe is daily.", True
+
+
+class FactorRowCountEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str, object]:
+        gt_df, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return (
+                "The source dataframe is None. Please check the implementation.",
+                False,
+            )
+        ratio = min(len(gen_df), len(gt_df)) / max(len(gen_df), len(gt_df))
+        return (
+            (
+                f"The ratio of rows count in the source dataframe to the ground truth dataframe is {ratio:.2f}. "
+                + "Please verify the implementation. "
+                if ratio <= 0.99
+                else ""
+            ),
+            ratio,
+        )
+
+
+class FactorIndexEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str, object]:
+        gt_df, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return (
+                "The source dataframe is None. Please check the implementation.",
+                False,
+            )
+        gen_index_set, gt_index_set = set(gen_df.index), set(gt_df.index)
+        similarity = len(gen_index_set.intersection(gt_index_set)) / len(gen_index_set.union(gt_index_set))
+        return (
+            (
+                f"The source dataframe and the ground truth dataframe have different index with a similarity of {similarity:.2%}. The similarity is calculated by the number of shared indices divided by the union indices. "
+                + "Please check the implementation."
+                if similarity <= 0.99
+                else ""
+            ),
+            similarity,
+        )
+
+
+class FactorMissingValuesEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str, object]:
+        gt_df, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return (
+                "The source dataframe is None. Please check the implementation.",
+                False,
+            )
+        if gen_df.isna().sum().sum() == gt_df.isna().sum().sum():
+            return "Both dataframes have the same missing values.", True
+        else:
+            return (
+                f"The dataframes do not have the same missing values. The source dataframe has {gen_df.isna().sum().sum()} missing values, while the ground truth dataframe has {gt_df.isna().sum().sum()} missing values. Please check the implementation.",
+                False,
+            )
+
+
+class FactorEqualValueRatioEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str, object]:
+        gt_df, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return (
+                "The source dataframe is None. Please check the implementation.",
+                -1,
+            )
+        try:
+            close_values = gen_df.sub(gt_df).abs().lt(1e-6)
+            result_int = close_values.astype(int)
+            pos_num = result_int.sum().sum()
+            acc_rate = pos_num / close_values.size
+        except:
+            close_values = gen_df
+        if close_values.all().iloc[0]:
+            return (
+                "All values in the dataframes are equal within the tolerance of 1e-6.",
+                acc_rate,
+            )
+        else:
+            return (
+                "Some values differ by more than the tolerance of 1e-6. Check for rounding errors or differences in the calculation methods.",
+                acc_rate,
+            )
+
+
+class FactorCorrelationEvaluator(FactorEvaluator):
+    def __init__(self, hard_check: bool, *args, **kwargs) -> None:
+        super().__init__(*args, **kwargs)
+        self.hard_check = hard_check
+
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+    ) -> Tuple[str, object]:
+        gt_df, gen_df = self._get_df(gt_implementation, implementation)
+        if gen_df is None:
+            return (
+                "The source dataframe is None. Please check the implementation.",
+                False,
+            )
+        concat_df = pd.concat([gen_df, gt_df], axis=1)
+        concat_df.columns = ["source", "gt"]
+        ic = concat_df.groupby("datetime").apply(lambda df: df["source"].corr(df["gt"])).dropna().mean()
+        ric = (
+            concat_df.groupby("datetime")
+            .apply(lambda df: df["source"].corr(df["gt"], method="spearman"))
+            .dropna()
+            .mean()
+        )
+
+        if self.hard_check:
+            if ic > 0.99 and ric > 0.99:
+                return (
+                    f"The dataframes are highly correlated. The ic is {ic:.6f} and the rankic is {ric:.6f}.",
+                    True,
+                )
+            else:
+                return (
+                    f"The dataframes are not sufficiently high correlated. The ic is {ic:.6f} and the rankic is {ric:.6f}. Investigate the factors that might be causing the discrepancies and ensure that the logic of the factor calculation is consistent.",
+                    False,
+                )
+        else:
+            return f"The ic is ({ic:.6f}) and the rankic is ({ric:.6f}).", ic
+
+
+class FactorValueEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        implementation: Workspace,
+        gt_implementation: Workspace,
+        version: int = 1,  # 1 for qlib factors and 2 for kaggle factors
+        **kwargs,
+    ) -> Tuple:
+        conclusions = []
+
+        # Initialize result variables
+        row_result = 0
+        index_result = 0
+        output_format_result = None
+        equal_value_ratio_result = 0
+        high_correlation_result = False
+        row_result = None
+
+        # Check if both dataframe has only one columns Mute this since factor task might generate more than one columns now
+        if version == 1:
+            feedback_str, _ = FactorSingleColumnEvaluator(self.scen).evaluate(implementation, gt_implementation)
+            conclusions.append(feedback_str)
+        elif version == 2:
+            input_shape = self.scen.input_shape
+            _, gen_df = self._get_df(gt_implementation, implementation)
+            if gen_df.shape[-1] > input_shape[-1]:
+                conclusions.append(
+                    "Output dataframe has more columns than input feature which is not acceptable in feature processing tasks. Please check the implementation to avoid generating too many columns. Consider this implementation as a failure."
+                )
+
+        feedback_str, inf_evaluate_res = FactorInfEvaluator(self.scen).evaluate(implementation, gt_implementation)
+        conclusions.append(feedback_str)
+
+        # Check if the index of the dataframe is ("datetime", "instrument")
+        feedback_str, _ = FactorOutputFormatEvaluator(self.scen).evaluate(implementation, gt_implementation)
+        conclusions.append(feedback_str)
+        if version == 1:
+            feedback_str, daily_check_result = FactorDatetimeDailyEvaluator(self.scen).evaluate(
+                implementation, gt_implementation
+            )
+            conclusions.append(feedback_str)
+        else:
+            daily_check_result = None
+
+        # Check dataframe format
+        if gt_implementation is not None:
+            feedback_str, row_result = FactorRowCountEvaluator(self.scen).evaluate(implementation, gt_implementation)
+            conclusions.append(feedback_str)
+
+            feedback_str, index_result = FactorIndexEvaluator(self.scen).evaluate(implementation, gt_implementation)
+            conclusions.append(feedback_str)
+
+            feedback_str, output_format_result = FactorMissingValuesEvaluator(self.scen).evaluate(
+                implementation, gt_implementation
+            )
+            conclusions.append(feedback_str)
+
+            feedback_str, equal_value_ratio_result = FactorEqualValueRatioEvaluator(self.scen).evaluate(
+                implementation, gt_implementation
+            )
+            conclusions.append(feedback_str)
+
+            if index_result > 0.99:
+                feedback_str, high_correlation_result = FactorCorrelationEvaluator(
+                    hard_check=True, scen=self.scen
+                ).evaluate(implementation, gt_implementation)
+            else:
+                high_correlation_result = False
+                feedback_str = "The source dataframe and the ground truth dataframe have different index. Give up comparing the values and correlation because it's useless"
+            conclusions.append(feedback_str)
+
+        # Combine all conclusions into a single string
+        conclusion_str = "\n".join(conclusions)
+
+        if gt_implementation is not None and (equal_value_ratio_result > 0.99) or high_correlation_result:
+            decision_from_value_check = True
+        elif (
+            row_result is not None
+            and row_result <= 0.99
+            or output_format_result is False
+            or daily_check_result is False
+            or inf_evaluate_res is False
+        ):
+            decision_from_value_check = False
+        else:
+            decision_from_value_check = None
+        return conclusion_str, decision_from_value_check
+
+
+class FactorFinalDecisionEvaluator(FactorEvaluator):
+    def evaluate(
+        self,
+        target_task: FactorTask,
+        execution_feedback: str,
+        value_feedback: str,
+        code_feedback: str,
+        **kwargs,
+    ) -> Tuple:
+        system_prompt = T(".prompts:evaluator_final_decision_v1_system").r(
+            scenario=(
+                self.scen.get_scenario_all_desc(target_task, filtered_tag="feature")
+                if self.scen is not None
+                else "No scenario description."
+            )
+        )
+        execution_feedback_to_render = execution_feedback
+
+        for _ in range(10):  # 10 times to split the content is enough
+            user_prompt = T(".prompts:evaluator_final_decision_v1_user").r(
+                factor_information=target_task.get_task_information(),
+                execution_feedback=execution_feedback_to_render,
+                code_feedback=code_feedback,
+                value_feedback=(
+                    value_feedback
+                    if value_feedback is not None
+                    else "No Ground Truth Value provided, so no evaluation on value is performed."
+                ),
+            )
+            if (
+                APIBackend().build_messages_and_calculate_token(
+                    user_prompt=user_prompt,
+                    system_prompt=system_prompt,
+                )
+                > APIBackend().chat_token_limit
+            ):
+                execution_feedback_to_render = execution_feedback_to_render[len(execution_feedback_to_render) // 2 :]
+            else:
+                break
+
+        # TODO:  with retry_context(retry_n=3, except_list=[KeyError]):
+        final_evaluation_dict = None
+        attempts = 0
+        max_attempts = 3
+
+        while attempts < max_attempts:
+            try:
+                api = APIBackend() if attempts == 0 else APIBackend(use_chat_cache=False)
+                final_evaluation_dict = json.loads(
+                    api.build_messages_and_create_chat_completion(
+                        user_prompt=user_prompt,
+                        system_prompt=system_prompt,
+                        json_mode=True,
+                        seed=attempts,  # in case of useless retrying when cache enabled.
+                        json_target_type=Dict[str, str | bool | int],
+                    ),
+                )
+                final_decision = final_evaluation_dict["final_decision"]
+                final_feedback = final_evaluation_dict["final_feedback"]
+
+                final_decision = str(final_decision).lower() in ["true", "1"]
+                return final_decision, final_feedback
+
+            except json.JSONDecodeError as e:
+                raise ValueError("Failed to decode JSON response from API.") from e
+            except KeyError as e:
+                attempts += 1
+                if attempts >= max_attempts:
+                    raise KeyError(
+                        "Response from API is missing 'final_decision' or 'final_feedback' key after multiple attempts."
+                    ) from e
+
+        return None, None

patches/factor_data_template_README.md

@@ -0,0 +1,36 @@
+# How to read files.
+For example, if you want to read `filename.h5`
+```Python
+import pandas as pd
+df = pd.read_hdf("filename.h5", key="data")
+```
+NOTE: **key is always "data" for all hdf5 files **.
+
+# Here is a short description about the data
+| Filename       | Description                                                      |
+| -------------- | -----------------------------------------------------------------|
+| "daily_pv.h5"  | EURUSD 15-minute OHLCV intraday data.                            |
+
+# For different data, We have some basic knowledge for them
+
+## EURUSD 15min intraday data
+$open: open price of EURUSD at the start of the 15min bar.
+$close: close price of EURUSD at the end of the 15min bar.
+$high: highest price of EURUSD during the 15min bar.
+$low: lowest price of EURUSD during the 15min bar.
+$volume: traded volume during the 15min bar.
+
+**IMPORTANT: There is NO $factor column. Use only $open, $close, $high, $low, $volume.**
+
+## Market sessions (UTC)
+- Asian session: 00:00 - 08:00 (mean reversion tendencies)
+- London session: 08:00 - 16:00 (trending, momentum works)
+- NY session: 13:00 - 21:00 (high volatility)
+- London-NY overlap: 13:00 - 16:00 (highest volume)
+
+## Lookback reference
+- 4 bars = 1 hour
+- 8 bars = 2 hours  
+- 16 bars = 4 hours
+- 32 bars = 8 hours
+- 96 bars = 1 day

patches/factor_proposal.py

@@ -0,0 +1,132 @@
+import json
+from typing import List, Tuple
+
+from rdagent.components.coder.factor_coder.factor import FactorExperiment, FactorTask
+from rdagent.components.proposal import FactorHypothesis2Experiment, FactorHypothesisGen
+from rdagent.core.proposal import Hypothesis, Scenario, Trace
+from rdagent.scenarios.qlib.experiment.factor_experiment import QlibFactorExperiment
+from rdagent.scenarios.qlib.experiment.model_experiment import QlibModelExperiment
+from rdagent.scenarios.qlib.experiment.quant_experiment import QlibQuantScenario
+from rdagent.utils.agent.tpl import T
+
+QlibFactorHypothesis = Hypothesis
+
+
+class QlibFactorHypothesisGen(FactorHypothesisGen):
+    def __init__(self, scen: Scenario) -> Tuple[dict, bool]:
+        super().__init__(scen)
+
+    def prepare_context(self, trace: Trace) -> Tuple[dict, bool]:
+        hypothesis_and_feedback = (
+            T("scenarios.qlib.prompts:hypothesis_and_feedback").r(
+                trace=trace,
+            )
+            if len(trace.hist) > 0
+            else "No previous hypothesis and feedback available since it's the first round."
+        )
+        last_hypothesis_and_feedback = (
+            T("scenarios.qlib.prompts:last_hypothesis_and_feedback").r(
+                experiment=trace.hist[-1][0], feedback=trace.hist[-1][1]
+            )
+            if len(trace.hist) > 0
+            else "No previous hypothesis and feedback available since it's the first round."
+        )
+
+        context_dict = {
+            "hypothesis_and_feedback": hypothesis_and_feedback,
+            "last_hypothesis_and_feedback": last_hypothesis_and_feedback,
+            "RAG": (
+                "Try EURUSD-specific FX factors: momentum (4-32 bars), mean reversion, ATR volatility, volume spikes, session-based signals. Use only $open $close $high $low $volume columns. No $factor column exists."
+                if len(trace.hist) < 15
+                else "Now, you need to try factors that can achieve high IC (e.g., machine learning-based factors)."
+            ),
+            "hypothesis_output_format": T("scenarios.qlib.prompts:factor_hypothesis_output_format").r(),
+            "hypothesis_specification": T("scenarios.qlib.prompts:factor_hypothesis_specification").r(),
+        }
+        return context_dict, True
+
+    def convert_response(self, response: str) -> Hypothesis:
+        response_dict = json.loads(response)
+        hypothesis = QlibFactorHypothesis(
+            hypothesis=response_dict.get("hypothesis"),
+            reason=response_dict.get("reason"),
+            concise_reason=response_dict.get("concise_reason"),
+            concise_observation=response_dict.get("concise_observation"),
+            concise_justification=response_dict.get("concise_justification"),
+            concise_knowledge=response_dict.get("concise_knowledge"),
+        )
+        return hypothesis
+
+
+class QlibFactorHypothesis2Experiment(FactorHypothesis2Experiment):
+    def prepare_context(self, hypothesis: Hypothesis, trace: Trace) -> Tuple[dict | bool]:
+        if isinstance(trace.scen, QlibQuantScenario):
+            scenario = trace.scen.get_scenario_all_desc(action="factor")
+        else:
+            scenario = trace.scen.get_scenario_all_desc()
+
+        experiment_output_format = T("scenarios.qlib.prompts:factor_experiment_output_format").r()
+
+        if len(trace.hist) == 0:
+            hypothesis_and_feedback = "No previous hypothesis and feedback available since it's the first round."
+        else:
+            specific_trace = Trace(trace.scen)
+            for i in range(len(trace.hist) - 1, -1, -1):
+                if not hasattr(trace.hist[i][0].hypothesis, "action") or trace.hist[i][0].hypothesis.action == "factor":
+                    specific_trace.hist.insert(0, trace.hist[i])
+            if len(specific_trace.hist) > 0:
+                specific_trace.hist.reverse()
+                hypothesis_and_feedback = T("scenarios.qlib.prompts:hypothesis_and_feedback").r(
+                    trace=specific_trace,
+                )
+            else:
+                hypothesis_and_feedback = "No previous hypothesis and feedback available."
+
+        return {
+            "target_hypothesis": str(hypothesis),
+            "scenario": scenario,
+            "hypothesis_and_feedback": hypothesis_and_feedback,
+            "experiment_output_format": experiment_output_format,
+            "target_list": [],
+            "RAG": None,
+        }, True
+
+    def convert_response(self, response: str, hypothesis: Hypothesis, trace: Trace) -> FactorExperiment:
+        response_dict = json.loads(response)
+        tasks = []
+
+        for factor_name in response_dict:
+            description = response_dict[factor_name]["description"]
+            formulation = response_dict[factor_name]["formulation"]
+            variables = response_dict[factor_name]["variables"]
+            tasks.append(
+                FactorTask(
+                    factor_name=factor_name,
+                    factor_description=description,
+                    factor_formulation=formulation,
+                    variables=variables,
+                )
+            )
+
+        exp = QlibFactorExperiment(tasks, hypothesis=hypothesis)
+        exp.based_experiments = [QlibFactorExperiment(sub_tasks=[])] + [
+            t[0] for t in trace.hist if t[1] and isinstance(t[0], FactorExperiment)
+        ]
+
+        unique_tasks = []
+        for task in tasks:
+            duplicate = False
+            for based_exp in exp.based_experiments:
+                if isinstance(based_exp, QlibModelExperiment):
+                    continue
+                for sub_task in based_exp.sub_tasks:
+                    if task.factor_name == sub_task.factor_name:
+                        duplicate = True
+                        break
+                if duplicate:
+                    break
+            if not duplicate:
+                unique_tasks.append(task)
+
+        exp.tasks = unique_tasks
+        return exp

patches/generate.py

@@ -0,0 +1,21 @@
+import subprocess
+import sys
+import os
+
+# Qlib läuft in rdagent4qlib environment
+result = subprocess.run(
+    ["/home/nico/miniconda3/envs/rdagent4qlib/bin/python3", "-c", """
+import qlib
+from qlib.data import D
+qlib.init(provider_uri="~/.qlib/qlib_data/eurusd_1min_data")
+fields = ["$open", "$close", "$high", "$low", "$volume"]
+data = (D.features(["EURUSD"], fields, start_time="2022-03-14", end_time="2026-03-20", freq="1min")
+        .swaplevel().sort_index())
+data.to_hdf("./daily_pv_all.h5", key="data")
+data_debug = (D.features(["EURUSD"], fields, start_time="2024-01-01", end_time="2026-03-20", freq="1min")
+              .swaplevel().sort_index())
+data_debug.to_hdf("./daily_pv_debug.h5", key="data")
+print(f"Done: {data.shape[0]} rows")
+"""],
+    capture_output=False
+)

patches/qlib_experiment_prompts.yaml

@@ -0,0 +1,257 @@
+qlib_quant_background: |-
+  Quantitative investment is a data-driven approach to asset management that relies on mathematical models, statistical techniques, and computational methods to analyze financial markets and make investment decisions. Two essential components of this approach are factors and models.
+  
+  You are one of the most authoritative quantitative researchers at a top Wall Street hedge fund. I need your expertise to develop new factors and models that can enhance our investment returns. Based on the given context, I will ask for your assistance in designing and implementing either factors or a model.
+
+  {% if runtime_environment is not none %}
+  ====== Runtime Environment ======
+  You have following environment to run the code:
+  {{ runtime_environment }}
+  {% endif %}
+
+qlib_factor_background: |-
+  The factor is a characteristic or variable used in quant investment that can help explain the returns and risks of a portfolio or a single asset. Factors are used by investors to identify and exploit sources of excess returns, and they are central to many quantitative investment strategies.
+  Each number in the factor represents a physics value to an instrument on a day.
+  User will train a model to predict the next several days return based on the factor values of the previous days.
+  The factor is defined in the following parts:
+  1. Name: The name of the factor.
+  2. Description: The description of the factor.
+  3. Formulation: The formulation of the factor.
+  4. Variables: The variables or functions used in the formulation of the factor.
+  The factor might not provide all the parts of the information above since some might not be applicable.
+  Please specifically give all the hyperparameter in the factors like the window size, look back period, and so on. One factor should statically defines one output with a static source data. For example, last 10 days momentum and last 20 days momentum should be two different factors.
+
+  {% if runtime_environment is not none %}
+  ====== Runtime Environment ======
+  You have following environment to run the code:
+  {{ runtime_environment }}
+  {% endif %}
+
+qlib_factor_interface: |-
+  Your python code should follow the interface to better interact with the user's system.
+  CRITICAL DATA FORMAT: The HDF5 file has a MultiIndex with levels ['datetime', 'instrument']. The instrument is an INDEX LEVEL, NOT a column. Never use df['instrument']. Always use df.index.get_level_values('instrument') or df.groupby(level='instrument'). For rolling calculations use df['$close'].unstack(level='instrument'), apply rolling, then .stack() to restore MultiIndex.
+  Your python code should contain the following part: the import part, the function part, and the main part. You should write a main function name: "calculate_{function_name}" and call this function in "if __name__ == __main__" part. Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you.
+  User will write your python code into a python file and execute the file directly with "python {your_file_name}.py". You should calculate the factor values and save the result into a HDF5(H5) file named "result.h5" in the same directory as your python file. The result file is a HDF5(H5) file containing a pandas dataframe. The index of the dataframe is the "datetime" and "instrument", and the single column name is the factor name,and the value is the factor value. The result file should be saved in the same directory as your python file.
+
+qlib_factor_strategy: |-
+  Ensure that for every step of data processing, the data format (including indexes) is clearly explained through comments.
+  Each transformation or calculation should be accompanied by a detailed description of how the data is structured, especially focusing on key aspects like whether the data has multi-level indexing, how to access specific columns or index levels, and any operations that affect the data shape (e.g., `reset_index()`, `groupby()`, `merge()`).
+  This step-by-step explanation will ensure clarity and accuracy in data handling. For example:
+  1. **Start with multi-level index**:  
+    ```python
+    # The initial DataFrame has a multi-level index with 'datetime' and 'instrument'.
+    # To access the 'datetime' index, use df.index.get_level_values('datetime').
+    datetime_values = df.index.get_level_values('datetime')
+    ```
+ 
+  2. **Reset the index if necessary**:  
+    ```python
+    # Resetting the index to move 'datetime' and 'instrument' from the index to columns.
+    # This operation flattens the multi-index structure.
+    df = df.reset_index()
+    ```
+ 
+  3. **Perform groupby operations**:  
+    ```python
+    # Grouping by 'datetime' and 'instrument' to aggregate the data.
+    # After groupby, the result will maintain 'datetime' and 'instrument' as a multi-level index.
+    df_grouped = df.groupby(['datetime', 'instrument']).sum()
+    ```
+ 
+  4. **Ensure consistent datetime formats**:  
+    ```python
+    # Before merging, ensure that the 'datetime' column in both DataFrames is of the same format.
+    # Convert to datetime format if necessary.
+    df['datetime'] = pd.to_datetime(df['datetime'])
+    other_df['datetime'] = pd.to_datetime(other_df['datetime'])
+    ```
+ 
+  5. **Merge operations**:  
+    ```python
+    # When merging DataFrames, ensure you are merging on both 'datetime' and 'instrument'.
+    # If these are part of the index, reset the index before merging.
+    merged_df = pd.merge(df, other_df, on=['datetime', 'instrument'], how='inner')
+    ```
+
+qlib_factor_output_format: |-
+  Your output should be a pandas dataframe similar to the following example information:
+  <class 'pandas.core.frame.DataFrame'>
+  MultiIndex: 2261923 entries, (Timestamp('2020-01-01 17:00:00'), 'EURUSD') to (Timestamp('2026-03-20 15:58:00'), 'EURUSD')
+  Data columns (total 1 columns):
+  #   Column            Non-Null Count  Dtype  
+  ---  ------            --------------  -----  
+  0   your factor name  2261923 non-null  float64
+  dtypes: float64(1)
+  memory usage: <ignore>
+  Notice: The non-null count is OK to be different to the total number of entries since some instruments may not have the factor value on some days.
+  One possible format of `result.h5` may be like following:
+  datetime    instrument
+  2020-01-01  EURUSD    1.094240
+  2020-01-02  EURUSD    1.094280
+  2020-01-03  EURUSD    1.095920
+                ...
+  2026-03-20  EURUSD    1.083150
+
+qlib_factor_simulator: |-
+  The factors will be sent into Qlib to train a model to predict the next several days return based on the factor values of the previous days. 
+  Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.
+  User will use Qlib to automatically do the following things:
+  1. generate a new factor table based on the factor values.
+  2. train a model like LightGBM, CatBoost, LSTM or simple PyTorch model to predict the next several days return based on the factor values.
+  3. build a portfolio based on the predicted return based on a strategy.
+  4. evaluate the portfolio's performance including the return, sharpe ratio, max drawdown, and so on.
+
+qlib_factor_rich_style_description : |-
+  ### R&D Agent-Qlib: Automated Quantitative Trading & Iterative Factors Evolution Demo
+
+  #### [Overview](#_summary)
+
+  The demo showcases the iterative process of hypothesis generation, knowledge construction, and decision-making. It highlights how financial factors evolve through continuous feedback and refinement.
+
+  #### [Automated R&D](#_rdloops)
+
+  - **[R (Research)](#_research)**
+    - Iterative development of ideas and hypotheses.
+    - Continuous learning and knowledge construction.
+
+  - **[D (Development)](#_development)**
+    - Progressive implementation and code generation of factors.
+    - Automated testing and validation of financial factors.
+
+  #### [Objective](#_summary)
+
+  To demonstrate the dynamic evolution of financial factors through the Qlib platform, emphasizing how each iteration enhances the accuracy and reliability of the resulting financial factors.
+
+qlib_factor_from_report_rich_style_description : |-
+  ### R&D Agent-Qlib: Automated Quantitative Trading & Factor Extraction from Financial Reports Demo
+
+  #### [Overview](#_summary)
+
+  This demo showcases the process of extracting factors from financial research reports, implementing these factors, and analyzing their performance through Qlib backtest, continually expanding and refining the factor library.
+
+  #### [Automated R&D](#_rdloops)
+
+  - **[R (Research)](#_research)**
+    - Iterative development of ideas and hypotheses from financial reports.
+    - Continuous learning and knowledge construction.
+
+  - **[D (Development)](#_development)**
+    - Progressive factor extraction and code generation.
+    - Automated implementation and testing of financial factors.
+
+  #### [Objective](#_summary)
+
+  <table border="1" style="width:100%; border-collapse: collapse;">
+    <tr>
+      <td>💡 <strong>Innovation </strong></td>
+      <td>Tool to quickly extract and test factors from research reports.</td>
+    </tr>
+    <tr>
+      <td>⚡ <strong>Efficiency </strong></td>
+      <td>Rapid identification of valuable factors from numerous reports.</td>
+    </tr>
+    <tr>
+      <td>🗃️ <strong>Outputs </strong></td>
+      <td>Expand and refine the factor library to support further research.</td>
+    </tr>
+  </table>
+
+qlib_factor_experiment_setting: |-
+  | Dataset 📊 | Model 🤖    | Factors 🌟       | Data Split  🧮                                   |
+  |---------|----------|---------------|-------------------------------------------------|
+  | EURUSD  | LGBModel | Alpha158 Plus | Train: 2022-01-01 to 2024-06-30 <br> Valid: 2024-07-01 to 2024-12-31 <br> Test &nbsp;: 2025-01-01 to 2026-03-20 |
+
+
+qlib_model_background: |-
+  The model is a machine learning or deep learning structure used in quantitative investment to predict the returns and risks of a portfolio or a single asset. Models are employed by investors to generate forecasts based on historical data and identified factors, which are central to many quantitative investment strategies.
+  Each model takes the factors as input and predicts the future returns. Usually, the bigger the model is, the better the performance would be.
+  The model is defined in the following parts:
+  1. Name: The name of the model.
+  2. Description: The description of the model.
+  3. Architecture: The detailed architecture of the model, such as neural network layers or tree structures.
+  4. Hyperparameters: The hyperparameters used in the model.
+  5. Training_hyperparameters: The hyperparameters used during the training process.
+  6. ModelType: The type of the model, "Tabular" for tabular model and "TimeSeries" for time series model.
+  The model should provide clear and detailed documentation of its architecture and hyperparameters. One model should statically define one output with a fixed architecture and hyperparameters.
+
+  {% if runtime_environment is not none %}
+  ====== Runtime Environment ======
+  You have following environment to run the code:
+  {{ runtime_environment }}
+  {% endif %}
+
+qlib_model_interface: |-
+  Your python code should follow the interface to better interact with the user's system.
+  You code should contain several parts:
+  1. The import part: import the necessary libraries.
+  2. A class which is a sub-class of pytorch.nn.Module. This class should should have a init function and a forward function which inputs a tensor and outputs a tensor.
+  3. Set a variable called "model_cls" to the class you defined.
+
+  The user will save your code into a python file called "model.py". Then the user imports model_cls in file "model.py" after setting the cwd into the directory:
+  ```python
+  from model import model_cls
+  ```
+  So your python code should follow the pattern:
+  ```python
+  class XXXModel(torch.nn.Module):
+      ...
+  model_cls = XXXModel
+  ```
+
+  The model can be configured as either "Tabular" for tabular models or "TimeSeries" for time series models. For a tabular model, the input shape is (batch_size, num_features), while for a time series model, the input shape is (batch_size, num_timesteps, num_features). In both cases, the output shape of the model should be (batch_size, 1).
+  `num_features` will be directly set for the model based on the input data shape.
+  User will initialize the tabular model with the following code:
+  ```python
+  model = model_cls(num_features=num_features)
+  ```
+  User will initialize the time series model with the following code:
+  ```python
+  model = model_cls(num_features=num_features, num_timesteps=num_timesteps)
+  ```
+  No other parameters will be passed to the model so give other parameters a default value or just make them static.
+
+  Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you. Also, don't write main function in your python code. The user will call the forward method in the model_cls to get the output tensor.
+
+  Please notice that your model should only use current features as input. The user will provide the input tensor to the model's forward function.
+
+
+qlib_model_output_format: |-
+  Your output should be a tensor with shape (batch_size, 1). 
+  The output tensor should be saved in a file named "output.pth" in the same directory as your python file.
+  The user will evaluate the shape of the output tensor so the tensor read from "output.pth" should be 8 numbers.
+
+qlib_model_simulator: |-
+  The models will be sent into Qlib to train and evaluate their performance in predicting future returns. Hypothesis is improved upon checking the feedback on the results. 
+  Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning (RL).
+  User will use Qlib to automatically perform the following tasks:
+  1. Generate a baseline factor table.
+  2. Train the model defined in your class Net to predict the next several days' returns based on the factor values.
+  3. Build a portfolio based on the predicted returns using a specific strategy.
+  4. Evaluate the portfolio's performance, including metrics such as return, IC, max drawdown, and others.
+  5. Iterate on growing the hypothesis to enable model improvements based on performance evaluations and feedback.
+
+qlib_model_rich_style_description: |-
+  ### Qlib Model Evolving Automatic R&D Demo
+  
+  #### [Overview](#_summary)
+  
+  The demo showcases the iterative process of hypothesis generation, knowledge construction, and decision-making in model construction in quantitative finance. It highlights how models evolve through continuous feedback and refinement.
+  
+  #### [Automated R&D](#_rdloops)
+  
+  - **[R (Research)](#_research)**
+    - Iteration of ideas and hypotheses.
+    - Continuous learning and knowledge construction.
+  
+  - **[D (Development)](#_development)**
+    - Evolving code generation and model refinement.
+    - Automated implementation and testing of models.
+  
+  #### [Objective](#_summary)
+  
+  To demonstrate the dynamic evolution of models through the Qlib platform, emphasizing how each iteration enhances the accuracy and reliability of the resulting models. 
+
+qlib_model_experiment_setting: |-
+  | Dataset 📊 | Model 🤖    | Factors 🌟       | Data Split  🧮                                   |
+  |---------|----------|---------------|-------------------------------------------------|
+  | EURUSD  | RDAgent-dev | 20 factors (Alpha158)  | Train: 2022-01-01 to 2024-06-30 <br> Valid: 2024-07-01 to 2024-12-31 <br> Test &nbsp;: 2025-01-01 to 2026-03-20 |

patches/qlib_rd_loop_prompts.yaml

@@ -0,0 +1,23 @@
+hypothesis_generation:
+  system: |-
+    You are an expert in FX and quantitative trading, specialized in EURUSD intraday strategies.
+    Your task is to generate a well-reasoned hypothesis for new alpha factors based on EURUSD 1min OHLCV data.
+    
+    Key market knowledge:
+    - EURUSD trades 24h with three main sessions: Asian (00:00-08:00 UTC), London (08:00-16:00 UTC), NY (13:00-21:00 UTC)
+    - London-NY overlap (13:00-16:00 UTC) has highest volume and momentum
+    - Asian session shows mean reversion tendencies
+    - Spread costs approximately 1.5 bps per trade — avoid overtrading
+    - No overnight gap risk like stocks, but weekend gaps exist
+    - Volume spikes signal news events (NFP, ECB, Fed)
+    
+    Please ensure your response is in JSON format as shown below:
+    {
+      "hypothesis": "A clear and concise hypothesis based on the provided information.",
+      "reason": "A detailed explanation supporting the generated hypothesis.",
+    }
+  user: |-
+    The following are the financial factors and their descriptions:
+    {{ factor_descriptions }}
+    The report content is as follows:
+    {{ report_content }}

patches/qlib_scenarios_prompts.yaml

@@ -0,0 +1,311 @@
+hypothesis_and_feedback: |-
+  =========================================================
+  {% for experiment, feedback in trace.hist %}
+  # Trial {{ loop.index }}: 
+  ## Hypothesis
+  {{ experiment.hypothesis }}
+  ## Specific task: 
+  {% for task in experiment.sub_tasks %}
+  {% if task is not none and task.get_task_brief_information is defined %}
+  {{ task.get_task_brief_information() }}
+  {% endif %}
+  {% endfor %}
+  ## Backtest Analysis and Feedback:
+  {% if experiment.result is not none %}
+  Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
+  {% endif %}
+  Observation: {{ feedback.observations }}
+  Hypothesis Evaluation: {{ feedback.hypothesis_evaluation }}
+  Decision (Whether the hypothesis was successful): {{ feedback.decision }}
+  =========================================================
+  {% endfor %}
+
+last_hypothesis_and_feedback: |-
+  ## Hypothesis
+  {{ experiment.hypothesis }}
+  ## Specific task: 
+  {% for task in experiment.sub_tasks %}
+  {% if task is not none and task.get_task_brief_information is defined %}
+  {{ task.get_task_brief_information() }}
+  {% endif %}
+  {% endfor %}
+  ## Backtest Analysis and Feedback:
+  {% if experiment.result is not none %}
+  Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
+  {% endif %}
+  Training Log: 
+  Here, you need to focus on analyzing whether there are any issues with the training. If any problems are identified, you must correct them in the next iteration and clearly describe how the changes will be made in the hypothesis.
+  {{ experiment.stdout }} 
+  Observation: {{ feedback.observations }}
+  Evaluation: {{ feedback.hypothesis_evaluation }}
+  Decision (Whether this experiment is SOTA): {{ feedback.decision }}
+  New Hypothesis (Given in feedback stage, just for reference, and can be accepted or rejected in the next round): {{ feedback.new_hypothesis }}
+  Reasoning (Justification for the new hypothesis): {{ feedback.reason }}
+
+sota_hypothesis_and_feedback: |-
+  ## Hypothesis
+  {{ experiment.hypothesis }}
+  ## Specific task: 
+  {% for task in experiment.sub_tasks %}
+  {% if task is not none and task.get_task_brief_information is defined %}
+  {{ task.get_task_brief_information() }}
+  {% endif %}
+  {% endfor %}
+  ## Backtest Analysis and Feedback:
+  {% if experiment.result is not none %}
+  Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
+  {% endif %}
+  Training Log: {{ experiment.stdout }}
+  Observation: {{ feedback.observations }}
+  Evaluation: {{ feedback.hypothesis_evaluation }}
+  Decision (Whether this experiment is SOTA): {{ feedback.decision }}
+
+hypothesis_output_format: |-
+  The output should follow JSON format. The schema is as follows:
+  {
+  "hypothesis": "An exact, testable, and innovative statement derived from previous experimental trace analysis. Avoid overly general ideas and ensure precision. The hypothesis should clearly specify the exact approach and expected improvement in performance in two or three sentences.",
+  "reason": "Provide a clear, logical explanation for why this hypothesis was proposed, grounded in evidence (e.g., trace history, domain principles). Reason should be short with no more than two sentences.",
+  }
+
+factor_hypothesis_output_format: |-
+  The output should follow JSON format. The schema is as follows:
+  {
+  "hypothesis": "The new hypothesis generated based on the information provided. Limit in two or three sentences.",
+  "reason": "The reason why you generate this hypothesis. It should be comprehensive and logical. It should cover the other keys below and extend them. Limit in two or three sentences.",
+  }
+
+hypothesis_output_format_with_action: |-
+  The output should follow JSON format. The schema is as follows:
+  {
+  "action": "If `hypothesis_specification` provides the action you need to take, please follow "hypothesis_specification" to choose the action. Otherwise, based on previous experimental results, suggest the action you believe is most appropriate at the moment. It should be one of [`factor`, `model`].",
+  "hypothesis": "The new hypothesis generated based on the information provided,should be a string.",
+  "reason": "The reason why you generate this hypothesis. It should be comprehensive and logical. It should cover the other keys below and extend them. Limit in two or three sentences.",
+  }
+
+model_hypothesis_specification: |-
+  1. First, observe and analyze the overall experimental progression in `hypothesis_and_feedback`. Analyze where the previous model designs were inadequate — whether it was due to parameter settings, architectural flaws, or a lack of novelty (proposing entirely new concepts is highly encouraged as long as they demonstrate effectiveness).
+  2. Second, `last_hypothesis_and_feedback` and `sota_hypothesis_and_feedback` are key references you should pay close attention to. You can choose to optimize based on either of them or generate new ideas to form hypotheses and experiments.
+  3. If there is no prior experiment or result available at the beginning, you can start by implementing a simple and small architecture.
+  4. If a series of attempts fail to achieve SOTA, consider exploring entirely new directions; at this point, it is acceptable to return to simple architectures.
+  5. Focus exclusively on the architecture of PyTorch models. Each hypothesis should specifically address architectural decisions, such as layer configurations, activation functions, regularization methods, and overall model structure. DO NOT do any feature-specific processing. Instead, you can propose innovative transformations on the input time-series data to enhance model training effectiveness.
+  6. Avoid including aspects unrelated to architecture, such as input features or optimization strategies.
+  7. Sometimes, when training performance is poor, adjusting hyperparameters can also be an effective strategy for improvement.
+  8. Use standard libraries for baseline models, but also explore custom architecture designs to investigate novel structures. After sufficient trials with traditional models, aim for innovation comparable to top-tier AI conferences (NeurIPS, ICLR, ICML, SIGKDD, etc.) in time series modeling.
+
+factor_hypothesis_specification: |-
+  You are developing alpha factors for EURUSD intraday trading using 15-minute OHLCV bars.
+
+  **Market Context:**
+  - EURUSD trades 24h with three sessions: Asian (00:00-08:00 UTC), London (08:00-16:00 UTC), NY (13:00-21:00 UTC)
+  - London-NY overlap (13:00-16:00 UTC) has highest volume and trending behavior
+  - Asian session shows mean reversion tendencies
+  - Spread cost ~1.5 bps per trade — avoid high-turnover factors
+  - No $factor column exists — use only $open, $close, $high, $low, $volume
+  - Each "instrument" is EURUSD, each "day" is a 1min bar
+
+  **Factor Generation Rules:**
+  1. **3-5 Factors per Generation** — cover different signal types per round
+  2. **FX-Specific Signals First:**
+    - Momentum: price change over last N bars (N=4,8,16,32 = 1h,2h,4h,8h)
+    - Mean Reversion: deviation from rolling mean, Bollinger Band position
+    - Volatility: ATR, realized vol, high-low range normalized
+    - Volume: volume spike ratio, volume trend
+    - Session: time-of-day encoded signals (London open, NY open)
+  3. **Gradual Complexity:**
+    - Rounds 1-5: single indicators (RSI, momentum, ATR)
+    - Rounds 6-15: combined signals (momentum + volume filter)
+    - Rounds 15+: ML-based factors (LSTM embeddings, XGBoost residuals)
+  4. **Avoid:**
+    - Factors requiring $factor column
+    - Daily-frequency assumptions (no overnight gaps in logic)
+    - Factors with >100 bar lookback without justification
+  5. No matter how many factors you plan to generate, only reply with one set of hypothesis and reason.
+
+factor_experiment_output_format: |-
+  The output should follow JSON format. The schema is as follows:
+  {
+      "factor name 1": {
+          "description": "description of factor 1, start with its type, e.g. [Momentum Factor]",
+          "formulation": "latex formulation of factor 1",
+          "variables": {
+              "variable or function name 1": "description of variable or function 1",
+              "variable or function name 2": "description of variable or function 2"
+          }
+      },
+      "factor name 2": {
+          "description": "description of factor 2, start with its type, e.g. [Machine Learning based Factor]",
+          "formulation": "latex formulation of factor 2",
+          "variables": {
+              "variable or function name 1": "description of variable or function 1",
+              "variable or function name 2": "description of variable or function 2"
+          }
+      }
+      # Don't add ellipsis (...) or any filler text that might cause JSON parsing errors here!
+  }
+
+model_experiment_output_format: |-
+  So far please only design one model to test the hypothesis! 
+  The output should follow JSON format. The schema is as follows (value in training_hyperparameters is a basic setting for reference, you CAN CHANGE depends on the previous training log): 
+  {
+    "model_name (The name of the model)": {
+        "description": "A detailed description of the model",
+        "formulation": "A LaTeX formula representing the model's formulation",
+        "architecture": "A detailed description of the model's architecture, e.g., neural network layers or tree structures",
+        "variables": {
+            "\\hat{y}_u": "The predicted output for node u",
+            "variable_name_2": "Description of variable 2",
+            "variable_name_3": "Description of variable 3"
+        },
+        "hyperparameters": {
+            "hyperparameter_name_1": "value of hyperparameter 1",
+            "hyperparameter_name_2": "value of hyperparameter 2",
+            "hyperparameter_name_3": "value of hyperparameter 3"
+        },
+        "training_hyperparameters" {  # All values are for reference; you can set them yourself
+            "n_epochs": "100",
+            "lr": "1e-3",
+            "early_stop": 10,
+            "batch_size": 256,
+            "weight_decay": 1e-4,
+        }
+        "model_type": "Tabular or TimeSeries"  # Should be one of "Tabular" or "TimeSeries"
+    },
+  }
+
+factor_feedback_generation:
+  system: |-
+    You are a professional FX quantitative analyst specializing in EURUSD intraday strategies.
+    The task is described in the following scenario:
+
+    {{ scenario }}
+
+    You will receive a hypothesis, multiple tasks with their factors, their results, and the SOTA result.
+    Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous SOTA results, and suggest FX-specific improvements.
+
+    **FX-specific evaluation criteria:**
+    - IC > 0.02 is meaningful for 1min EURUSD data
+    - Annualized return target: >9.62% (current SOTA to beat)
+    - Spread cost ~1.5 bps per trade — penalize high-turnover factors
+    - Factors using $factor column are INVALID — only $open $close $high $low $volume allowed
+    - Session-aware factors (London/NY) tend to outperform session-agnostic ones
+    - Mean reversion works in Asian session, momentum in London-NY overlap
+
+    Please understand the following operation logic:
+      1. Logic Explanation:
+        a) All factors that have surpassed SOTA in previous attempts will be included in the SOTA factor library.
+        b) New experiments will generate new factors, combined with the SOTA library factors.
+        c) These combined factors will be backtested and compared against current SOTA.
+      2. Development Directions:
+        a) New Direction: Propose a new FX-specific factor (session filter, volatility regime, volume spike).
+        b) Optimization: Refine lookback windows (4/8/16/32 bars), add ADX filter, adjust for spread costs.
+      3. Final Goal: Beat 9.62% ARR on EURUSD 1min with controlled drawdown (<20%).
+
+    When judging results:
+      1. Any small improvement in annualized return → set Replace Best Result as yes.
+      2. If IC < 0 consistently → factor has no predictive power, change direction entirely.
+      3. High turnover with low return → add volume or volatility filter to reduce trade frequency.
+
+    Respond in JSON format:
+    {
+      "Observations": "Your overall observations here",
+      "Feedback for Hypothesis": "Observations related to the hypothesis",
+      "New Hypothesis": "Your new FX-specific hypothesis here",
+      "Reasoning": "Reasoning for the new hypothesis",
+      "Replace Best Result": "yes or no"
+    }
+  user: |-
+    Target hypothesis: 
+    {{ hypothesis_text }}
+    Tasks and Factors:
+    {% for task in task_details %}
+      - {{ task.factor_name }}: {{ task.factor_description }}
+        - Factor Formulation: {{ task.factor_formulation }}
+        - Variables: {{ task.variables }}
+        - Factor Implementation: {{ task.factor_implementation }}
+        {% if task.factor_implementation == "False" %}
+        **Note: This factor was not implemented in the current experiment. Only the hypothesis for implemented factors can be verified.**
+        {% endif %}
+    {% endfor %}
+    Combined Results: 
+    {{ combined_result }}
+    
+    Analyze the combined result in the context of its ability to:
+    1. Support or refute the hypothesis.
+    2. Show improvement or deterioration compared to the SOTA experiment.
+    
+    Note: Only factors with 'Factor Implementation' as True are implemented and tested in this experiment. If 'Factor Implementation' is False, the hypothesis for that factor cannot be verified in this run.
+
+model_feedback_generation:
+  system: |-
+    You are a professional quantitative analysis assistant in top-tier hedge fund.
+
+    The task is described in the following scenario:
+    {{ scenario }}
+
+    You will receive a quantitative model hypothesis, its specific task description, and it market backtest result. 
+    Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous SOTA results, examine the model's training logs to analyze whether there are issues with hyperparameter settings, and suggest improvements or new directions.
+
+    Please provide detailed and constructive feedback.
+    Example JSON Structure for Result Analysis:
+    {
+      "Observations": "First analyze the model's training logs to determine whether there are any issues with its parameter settings. Then clearly summarize the current results and the SOTA results with exact scores and any notable patterns. Limit your summary to no more than three concise, data-focused sentences.",
+      "Feedback for Hypothesis": "Explicitly confirm or refute the hypothesis based on specific data points or performance trends. Limit to two sentences.",
+      "New Hypothesis": "Propose a revised hypothesis, considering observed patterns and limitations in the current one. Limit to no more than two sentences.",
+      "Reasoning": "Explain the rationale for the new hypothesis using specific trends or performance shifts. Be concise but technically complete. Limit to two sentences.",
+      "Decision": <true or false>,
+    }
+
+    
+  user: |-
+    {% if sota_hypothesis %} 
+    # SOTA Round Information:
+    Hypothesis: {{ sota_hypothesis.hypothesis }}
+    Specific Task: {{ sota_task }}
+    Code Implementation: {{ sota_code }}
+    Result: {{ sota_result }}
+    {% else %}
+    # This is the first round. No previous information available. As long as the performance is not too negative (eg.ICIR is greater than 0), treat it as successful. Do not set the threshold too high.  
+    {% endif %} 
+    
+    # Current Round Information:
+    Hypothesis: {{ hypothesis.hypothesis }}
+    Why propose this hypothesis: {{ hypothesis.reason }}
+    Specific Task: {{ exp.sub_tasks[0].get_task_information() }}
+    Code Implementation: {{ exp.sub_workspace_list[0].file_dict.get("model.py") }}
+    Training Log: {{ exp.stdout }}
+    Result: {{ exp_result }}
+
+    # When judging the results:
+    1. **Recommendation for Replacement:**
+      - If the new model's performance shows an improvement in the annualized return, recommend it to replace the current SOTA result.
+      - Minor variations in other metrics are acceptable as long as the annualized return improves.
+    2.  Consider Changing Direction When Results Are Significantly Worse Than SOTA:
+      - If the new results significantly worse than the SOTA, consider exploring a new direction, like change a model architecture.
+
+action_gen:
+  system: |-
+    Quantitative investment is a data-driven approach to asset management that relies on mathematical models, statistical techniques, and computational methods to analyze financial markets and make investment decisions. Two essential components of this approach are factors and models.
+  
+    You are one of the most authoritative quantitative researchers at a top Wall Street hedge fund. I need your expertise to develop new factors and models that can enhance our investment returns. Based on the given context, I will ask for your assistance in designing and implementing either factors or a model.
+
+    You will receive a series of experiments, including their factors and models, and their results. 
+    Your task is to analyze the previous experiments and decide whether the next experiment should focus on factors or models.
+
+    Example JSON Structure for your return:
+    {
+      "action": "factor" or "model",  # You must choose one of the two
+    }
+
+  user: |-
+    {% if hypothesis_and_feedback|length == 0 %}
+    It is the first round of hypothesis generation. The user has no hypothesis on this scenario yet.
+    {% else %}
+    The former hypothesis and the corresponding feedbacks are as follows:
+    {{ hypothesis_and_feedback }}
+    {% endif %}
+
+  
+    {% if last_hypothesis_and_feedback != "" %}
+    Here is the last trial's hypothesis and the corresponding feedback. The main feedback includes a new hypothesis for your reference only. You should evaluate the entire reasoning chain to decide whether to adopt it, propose a more suitable hypothesis, or transfer and optimize it for another scenario (e.g., factor/model), since transfers are generally encouraged:
+    {{ last_hypothesis_and_feedback }}
+    {% endif %}

pyproject.toml

@@ -7,7 +7,7 @@ requires = [
 
 [project]
 authors = [
-  {email = "xuyang1@microsoft.com", name = "MSRA-MIIC"},
+  {email = "nico@predix.io", name = "Predix Team"},
 ]
 classifiers = [
   "Development Status :: 3 - Alpha",
@@ -16,18 +16,20 @@ classifiers = [
   "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
 ]
-description = "Research & Development Agent"
+description = "Predix - AI-gestützter Quantitative Trading Agent für EUR/USD"
 dynamic = [
   "dependencies",
   "optional-dependencies",
   "version",
 ]
 keywords = [
+  "Quantitative Trading",
   "Autonomous Agents",
   "Large Language Models",
-  "Research and Development",
+  "EUR/USD",
+  "Forex",
 ]
-name = "rdagent"
+name = "predix"
 readme = "README.md"
 requires-python = ">=3.10"
 
@@ -35,8 +37,8 @@ requires-python = ">=3.10"
 rdagent = "rdagent.app.cli:app"
 
 [project.urls]
-homepage = "https://github.com/microsoft/RD-Agent/"
-issue = "https://github.com/microsoft/RD-Agent/issues"
+homepage = "https://github.com/PredixAI/predix/"
+issue = "https://github.com/PredixAI/predix/issues"
 
 [tool.coverage.report]
 fail_under = 80
@@ -71,7 +73,13 @@ log_cli = true
 log_cli_level = "info"
 log_date_format = "%Y-%m-%d %H:%M:%S"
 log_format = "%(asctime)s %(levelname)s %(message)s"
+markers = [
+  "offline: tests that do not require external API calls",
+]
 minversion = "6.0"
+norecursedirs = [
+  "workspace",
+]
 
 [tool.ruff]
 fix = true
@@ -112,6 +120,7 @@ docs = {file = ["requirements/docs.txt"]}
 lint = {file = ["requirements/lint.txt"]}
 package = {file = ["requirements/package.txt"]}
 test = {file = ["requirements/test.txt"]}
+torch = {file = ["requirements/torch.txt"]} # some agent algorithms need torch.  pip install rdagent[torch]
 
 [tool.setuptools_scm]
 local_scheme = "no-local-version"

rdagent/app/cli.py

@@ -6,6 +6,8 @@ This will
 - autoamtically load dotenv
 """
 
+import sys
+
 from dotenv import load_dotenv
 
 load_dotenv(".env")
@@ -14,10 +16,13 @@ load_dotenv(".env")
 
 import subprocess
 from importlib.resources import path as rpath
+from typing import Optional
 
-import fire
+import typer
+from typing_extensions import Annotated
 
 from rdagent.app.data_science.loop import main as data_science
+from rdagent.app.finetune.llm.loop import main as llm_finetune
 from rdagent.app.general_model.general_model import (
     extract_models_and_implement as general_model,
 )
@@ -27,12 +32,25 @@ from rdagent.app.qlib_rd_loop.model import main as fin_model
 from rdagent.app.qlib_rd_loop.quant import main as fin_quant
 from rdagent.app.utils.health_check import health_check
 from rdagent.app.utils.info import collect_info
+from rdagent.log.mle_summary import grade_summary as grade_summary
+
+app = typer.Typer()
+
+CheckoutOption = Annotated[bool, typer.Option("--checkout/--no-checkout", "-c/-C")]
+CheckEnvOption = Annotated[bool, typer.Option("--check-env/--no-check-env", "-e/-E")]
+CheckDockerOption = Annotated[bool, typer.Option("--check-docker/--no-check-docker", "-d/-D")]
+CheckPortsOption = Annotated[bool, typer.Option("--check-ports/--no-check-ports", "-p/-P")]
 
 
-def ui(port=19899, log_dir="", debug=False):
+def ui(port=19899, log_dir="", debug: bool = False, data_science: bool = False):
     """
     start web app to show the log traces.
     """
+    if data_science:
+        with rpath("rdagent.log.ui", "dsapp.py") as app_path:
+            cmds = ["streamlit", "run", app_path, f"--server.port={port}"]
+            subprocess.run(cmds)
+        return
     with rpath("rdagent.log.ui", "app.py") as app_path:
         cmds = ["streamlit", "run", app_path, f"--server.port={port}"]
         if log_dir or debug:
@@ -45,24 +63,141 @@ def ui(port=19899, log_dir="", debug=False):
 
 
 def server_ui(port=19899):
+    """
+    start the Flask log server in real time
+    """
+    from rdagent.log.server.app import main as log_server_main
+
+    log_server_main(port=port)
+
+
+def ds_user_interact(port=19900):
     """
     start web app to show the log traces in real time
     """
-    subprocess.run(["python", "rdagent/log/server/app.py", f"--port={port}"])
+    commands = ["streamlit", "run", "rdagent/log/ui/ds_user_interact.py", f"--server.port={port}"]
+    subprocess.run(commands)
 
 
-def app():
-    fire.Fire(
-        {
-            "fin_factor": fin_factor,
-            "fin_factor_report": fin_factor_report,
-            "fin_model": fin_model,
-            "fin_quant": fin_quant,
-            "general_model": general_model,
-            "ui": ui,
-            "health_check": health_check,
-            "collect_info": collect_info,
-            "data_science": data_science,
-            "server_ui": server_ui,
-        }
+@app.command(name="fin_factor")
+def fin_factor_cli(
+    path: Optional[str] = None,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    all_duration: Optional[str] = None,
+    checkout: CheckoutOption = True,
+):
+    fin_factor(path=path, step_n=step_n, loop_n=loop_n, all_duration=all_duration, checkout=checkout)
+
+
+@app.command(name="fin_model")
+def fin_model_cli(
+    path: Optional[str] = None,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    all_duration: Optional[str] = None,
+    checkout: CheckoutOption = True,
+):
+    fin_model(path=path, step_n=step_n, loop_n=loop_n, all_duration=all_duration, checkout=checkout)
+
+
+@app.command(name="fin_quant")
+def fin_quant_cli(
+    path: Optional[str] = None,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    all_duration: Optional[str] = None,
+    checkout: CheckoutOption = True,
+):
+    fin_quant(path=path, step_n=step_n, loop_n=loop_n, all_duration=all_duration, checkout=checkout)
+
+
+@app.command(name="fin_factor_report")
+def fin_factor_report_cli(
+    report_folder: Optional[str] = None,
+    path: Optional[str] = None,
+    all_duration: Optional[str] = None,
+    checkout: CheckoutOption = True,
+):
+    fin_factor_report(report_folder=report_folder, path=path, all_duration=all_duration, checkout=checkout)
+
+
+@app.command(name="general_model")
+def general_model_cli(report_file_path: str):
+    general_model(report_file_path)
+
+
+@app.command(name="data_science")
+def data_science_cli(
+    path: Optional[str] = None,
+    checkout: CheckoutOption = True,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    timeout: Optional[str] = None,
+    competition: Optional[str] = None,
+):
+    data_science(
+        path=path,
+        checkout=checkout,
+        step_n=step_n,
+        loop_n=loop_n,
+        timeout=timeout,
+        competition=competition,
     )
+
+
+@app.command(name="llm_finetune")
+def llm_finetune_cli(
+    path: Optional[str] = None,
+    checkout: CheckoutOption = True,
+    benchmark: Optional[str] = None,
+    benchmark_description: Optional[str] = None,
+    dataset: Optional[str] = None,
+    base_model: Optional[str] = None,
+    upper_data_size_limit: Optional[int] = None,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    timeout: Optional[str] = None,
+):
+    llm_finetune(
+        path=path,
+        checkout=checkout,
+        benchmark=benchmark,
+        benchmark_description=benchmark_description,
+        dataset=dataset,
+        base_model=base_model,
+        upper_data_size_limit=upper_data_size_limit,
+        step_n=step_n,
+        loop_n=loop_n,
+        timeout=timeout,
+    )
+
+
+@app.command(name="grade_summary")
+def grade_summary_cli(log_folder: str):
+    grade_summary(log_folder)
+
+
+app.command(name="ui")(ui)
+app.command(name="server_ui")(server_ui)
+
+
+@app.command(name="health_check")
+def health_check_cli(
+    check_env: CheckEnvOption = True,
+    check_docker: CheckDockerOption = True,
+    check_ports: CheckPortsOption = True,
+):
+    health_check(check_env=check_env, check_docker=check_docker, check_ports=check_ports)
+
+
+@app.command(name="collect_info")
+def collect_info_cli():
+    collect_info()
+
+
+app.command(name="ds_user_interact")(ds_user_interact)
+
+
+if __name__ == "__main__":
+    app()

rdagent/app/data_science/conf.py

@@ -1,3 +1,4 @@
+from pathlib import Path
 from typing import Literal
 
 from pydantic_settings import SettingsConfigDict
@@ -18,43 +19,61 @@ class DataScienceBasePropSetting(KaggleBasePropSetting):
     - For custom data science scenarios, use: "rdagent.scenarios.data_science.scen.DataScienceScen"
     """
 
-    hypothesis_gen: str = "rdagent.scenarios.data_science.proposal.exp_gen.proposal.DSProposalV2ExpGen"
+    planner: str = "rdagent.scenarios.data_science.proposal.exp_gen.planner.DSExpPlannerHandCraft"
+    hypothesis_gen: str = "rdagent.scenarios.data_science.proposal.exp_gen.router.ParallelMultiTraceExpGen"
+    interactor: str = "rdagent.components.interactor.SkipInteractor"
+    trace_scheduler: str = "rdagent.scenarios.data_science.proposal.exp_gen.trace_scheduler.RoundRobinScheduler"
     """Hypothesis generation class"""
 
+    summarizer: str = "rdagent.scenarios.data_science.dev.feedback.DSExperiment2Feedback"
+    summarizer_init_kwargs: dict = {
+        "version": "exp_feedback",
+    }
     ## Workflow Related
     consecutive_errors: int = 5
 
     ## Coding Related
     coding_fail_reanalyze_threshold: int = 3
 
+    debug_recommend_timeout: int = 600
+    """The recommend time limit for running on debugging data"""
     debug_timeout: int = 600
     """The timeout limit for running on debugging data"""
+    full_recommend_timeout: int = 3600
+    """The recommend time limit for running on full data"""
     full_timeout: int = 3600
     """The timeout limit for running on full data"""
 
-    ### specific feature
-
-    #### enable specification
-    spec_enabled: bool = True
-
-    #### proposal related
-    proposal_version: str = "v1"
-    coder_on_whole_pipeline: bool = False
-    max_trace_hist: int = 3
-
-    coder_max_loop: int = 10
-    runner_max_loop: int = 1
-
-    rule_base_eval: bool = False
-    sample_data: bool = True
-    use_raw_description: bool = False
-    show_nan_columns: bool = False
-
     #### model dump
     enable_model_dump: bool = False
     enable_doc_dev: bool = False
     model_dump_check_level: Literal["medium", "high"] = "medium"
 
+    #### MCP documentation search integration
+    enable_mcp_documentation_search: bool = False
+    """Enable MCP documentation search for error resolution. Requires MCP_ENABLED=true and MCP_CONTEXT7_ENABLED=true in environment."""
+
+    ### specific feature
+
+    ### notebook integration
+    enable_notebook_conversion: bool = False
+
+    #### enable specification
+    spec_enabled: bool = True
+
+    #### proposal related
+    # proposal_version: str = "v2" deprecated
+
+    coder_on_whole_pipeline: bool = True
+    max_trace_hist: int = 3
+
+    coder_max_loop: int = 10
+    runner_max_loop: int = 3
+
+    sample_data_by_LLM: bool = True
+    use_raw_description: bool = False
+    show_nan_columns: bool = False
+
     ### knowledge base
     enable_knowledge_base: bool = False
     knowledge_base_version: str = "v1"
@@ -76,11 +95,21 @@ class DataScienceBasePropSetting(KaggleBasePropSetting):
     """---below are the settings for multi-trace---"""
 
     ### multi-trace related
-    max_trace_num: int = 3
+    max_trace_num: int = 1
     """The maximum number of traces to grow before merging"""
 
+    scheduler_temperature: float = 1.0
+    """The temperature for the trace scheduler for softmax calculation, used in ProbabilisticScheduler"""
+
+    # PUCT exploration constant for MCTSScheduler (ignored by other schedulers)
+    scheduler_c_puct: float = 1.0
+    """Exploration constant used by MCTSScheduler (PUCT)."""
+
+    enable_score_reward: bool = False
+    """Enable using score-based reward for trace selection in multi-trace scheduling."""
+
     #### multi-trace:checkpoint selector
-    selector_name: str = "rdagent.scenarios.data_science.proposal.exp_gen.ckp_select.LatestCKPSelector"
+    selector_name: str = "rdagent.scenarios.data_science.proposal.exp_gen.select.expand.LatestCKPSelector"
     """The name of the selector to use"""
     sota_count_window: int = 5
     """The number of trials to consider for SOTA count"""
@@ -88,22 +117,29 @@ class DataScienceBasePropSetting(KaggleBasePropSetting):
     """The threshold for SOTA count"""
 
     #### multi-trace: SOTA experiment selector
-    sota_exp_selector_name: str = "rdagent.scenarios.data_science.proposal.exp_gen.sota_exp_select.GlobalSOTASelector"
+    sota_exp_selector_name: str = "rdagent.scenarios.data_science.proposal.exp_gen.select.submit.GlobalSOTASelector"
     """The name of the SOTA experiment selector to use"""
 
     ### multi-trace:inject optimals for multi-trace
     # inject diverse when start a new sub-trace
     enable_inject_diverse: bool = False
 
-    # inject knowledge at the root of the trace
-    enable_inject_knowledge_at_root: bool = False
+    # inject diverse from other traces when start a new sub-trace
+    enable_cross_trace_diversity: bool = True
+    """Enable cross-trace diversity injection when starting a new sub-trace.
+    This is different from `enable_inject_diverse` which is for non-parallel cases."""
+
+    diversity_injection_strategy: str = (
+        "rdagent.scenarios.data_science.proposal.exp_gen.diversity_strategy.InjectUntilSOTAGainedStrategy"
+    )
+    """The strategy to use for injecting diversity context."""
 
     # enable different version of DSExpGen for multi-trace
     enable_multi_version_exp_gen: bool = False
     exp_gen_version_list: str = "v3,v2"
 
     #### multi-trace: time for final multi-trace merge
-    merge_hours: int = 2
+    merge_hours: float = 0
     """The time for merge"""
 
     #### multi-trace: max SOTA-retrieved number, used in AutoSOTAexpSelector
@@ -111,5 +147,60 @@ class DataScienceBasePropSetting(KaggleBasePropSetting):
     max_sota_retrieved_num: int = 10
     """The maximum number of SOTA experiments to retrieve in a LLM call"""
 
+    #### enable draft before first sota experiment
+    enable_draft_before_first_sota: bool = False
+    enable_planner: bool = False
+
+    model_architecture_suggestion_time_percent: float = 0.75
+    allow_longer_timeout: bool = False
+    coder_enable_llm_decide_longer_timeout: bool = False
+    runner_enable_llm_decide_longer_timeout: bool = False
+    coder_longer_timeout_multiplier_upper: int = 3
+    runner_longer_timeout_multiplier_upper: int = 2
+    coder_timeout_increase_stage: float = 0.3
+    runner_timeout_increase_stage: float = 0.3
+    runner_timeout_increase_stage_patience: int = 2
+    """Number of failures tolerated before escalating to next timeout level (stage width). Every 'patience' failures, timeout increases by 'runner_timeout_increase_stage'"""
+    show_hard_limit: bool = True
+
+    #### enable runner code change summary
+    runner_enable_code_change_summary: bool = True
+
+    ### Proposal workflow related
+
+    #### Hypothesis Generate related
+    enable_simple_hypothesis: bool = False
+    """If true, generate simple hypothesis, no more than 2 sentences each."""
+
+    enable_generate_unique_hypothesis: bool = False
+    """Enable generate unique hypothesis. If True, generate unique hypothesis for each component. If False, generate unique hypothesis for each component."""
+
+    enable_research_rag: bool = False
+    """Enable research RAG for hypothesis generation."""
+
+    #### hypothesis critique and rewrite
+    enable_hypo_critique_rewrite: bool = False
+    """Enable hypothesis critique and rewrite stages for improving hypothesis quality"""
+    enable_scale_check: bool = False
+
+    ##### select related
+    ratio_merge_or_ensemble: int = 70
+    """The ratio of merge or ensemble to be considered as a valid solution"""
+    llm_select_hypothesis: bool = False
+    """Whether to use LLM to select hypothesis. If True, use LLM selection; if False, use the existing ranking method."""
+
+    #### Task Generate related
+    fix_seed_and_data_split: bool = False
+
+    ensemble_time_upper_bound: bool = False
+
+    user_interaction_wait_seconds: int = 6000  # seconds to wait for user interaction
+    user_interaction_mid_folder: Path = Path.cwd() / "git_ignore_folder" / "RD-Agent_user_interaction"
+
 
 DS_RD_SETTING = DataScienceBasePropSetting()
+
+# enable_cross_trace_diversity and llm_select_hypothesis should not be true at the same time
+assert not (
+    DS_RD_SETTING.enable_cross_trace_diversity and DS_RD_SETTING.llm_select_hypothesis
+), "enable_cross_trace_diversity and llm_select_hypothesis cannot be true at the same time"

rdagent/app/data_science/loop.py

@@ -1,5 +1,6 @@
 import asyncio
 from pathlib import Path
+from typing import Optional
 
 import fire
 
@@ -10,14 +11,15 @@ from rdagent.scenarios.data_science.loop import DataScienceRDLoop
 
 
 def main(
-    path: str | None = None,
-    checkout: bool | str | Path = True,
-    step_n: int | None = None,
-    loop_n: int | None = None,
+    path: Optional[str] = None,
+    checkout: bool = True,
+    checkout_path: Optional[str] = None,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    timeout: Optional[str] = None,
     competition="bms-molecular-translation",
-    timeout=None,
     replace_timer=True,
-    exp_gen_cls: str | None = None,
+    exp_gen_cls: Optional[str] = None,
 ):
     """
 
@@ -26,18 +28,20 @@ def main(
     path :
         A path like `$LOG_PATH/__session__/1/0_propose`. This indicates that we restore the state after finishing step 0 in loop 1.
     checkout :
-        Used only when a path is provided.
-        Can be True, False, or a path.
-        Default is True.
+        Used to control the log session path. Boolean type, default is True.
         - If True, the new loop will use the existing folder and clear logs for sessions after the one corresponding to the given path.
         - If False, the new loop will use the existing folder but keep the logs for sessions after the one corresponding to the given path.
-        - If a path (or a str like Path) is provided, the new loop will be saved to that path, leaving the original path unchanged.
+    checkout_path:
+        If a checkout_path (or a str like Path) is provided, the new loop will be saved to that path, leaving the original path unchanged.
     step_n :
         Number of steps to run; if None, the process will run indefinitely until an error or KeyboardInterrupt occurs.
     loop_n :
         Number of loops to run; if None, the process will run indefinitely until an error or KeyboardInterrupt occurs.
         - If the current loop is incomplete, it will be counted as the first loop for completion.
         - If both step_n and loop_n are provided, the process will stop as soon as either condition is met.
+    timeout :
+        Maximum duration to run the loop. Accepts a string format recognized by the internal timer.
+        - If None, the loop will run until completion, error, or KeyboardInterrupt.
     competition :
         Competition name.
     replace_timer :
@@ -48,10 +52,15 @@ def main(
 
     Auto R&D Evolving loop for models in a Kaggle scenario.
     You can continue running a session by using the command:
+
     .. code-block:: bash
-        dotenv run -- python rdagent/app/data_science/loop.py [--competition titanic] $LOG_PATH/__session__/1/0_propose  --step_n 1   # `step_n` is an optional parameter
-        rdagent kaggle --competition playground-series-s4e8  # This command is recommended.
+
+      dotenv run -- python rdagent/app/data_science/loop.py [--competition titanic] $LOG_PATH/__session__/1/0_propose  --step_n 1   # `step_n` is an optional parameter
+      rdagent kaggle --competition playground-series-s4e8  # This command is recommended.
     """
+    if not checkout_path is None:
+        checkout = Path(checkout_path)
+
     if competition is not None:
         DS_RD_SETTING.competition = competition
 

rdagent/app/finetune/data_science/conf.py

@@ -0,0 +1,40 @@
+import os
+
+from pydantic_settings import SettingsConfigDict
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.core.conf import RD_AGENT_SETTINGS, ExtendedBaseSettings
+
+
+class DSFinetuneScen(ExtendedBaseSettings):
+    model_config = SettingsConfigDict(env_prefix="FT_", protected_namespaces=())
+    scen: str = "rdagent.app.finetune.data_science.scen.DSFinetuneScen"
+    """
+    Scenario class for data science tasks.
+    - For Kaggle competitions, use: "rdagent.scenarios.data_science.scen.KaggleScen"
+    - For custom data science scenarios, use: "rdagent.scenarios.data_science.scen.DataScienceScen"
+    - For LLM finetune scenarios, use: "rdagent.app.finetune.llm.scen.LLMFinetuneScen"
+    - For Data science finetune scenarios, use: "rdagent.app.finetune.data_science.scen.DSFinetuneScen"
+    """
+
+    debug_timeout: int = 3600
+    """The timeout limit for running on debugging data"""
+    full_timeout: int = 10800
+    """The timeout limit for running on full data"""
+
+    coder_on_whole_pipeline: bool = True
+    enable_model_dump: bool = True
+    app_tpl: str = "app/finetune/data_science/tpl"
+
+
+def update_settings(competition: str):
+    """
+    Update the RD_AGENT_SETTINGS with the values from DS_FINETUNE_SETTINGS.
+    """
+    DS_FINETUNE_SETTINGS = DSFinetuneScen()
+    RD_AGENT_SETTINGS.app_tpl = DS_FINETUNE_SETTINGS.app_tpl
+    os.environ["DS_CODER_COSTEER_EXTRA_EVALUATOR"] = '["rdagent.app.finetune.share.eval.PrevModelLoadEvaluator"]'
+    for field_name, new_value in DS_FINETUNE_SETTINGS.model_dump().items():
+        if hasattr(DS_RD_SETTING, field_name):
+            setattr(DS_RD_SETTING, field_name, new_value)
+    DS_RD_SETTING.competition = competition

rdagent/app/finetune/data_science/loop.py

@@ -0,0 +1,40 @@
+import asyncio
+from pathlib import Path
+
+import fire
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.app.finetune.data_science.conf import update_settings
+from rdagent.core.utils import import_class
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.data_science.loop import DataScienceRDLoop
+
+
+def main(
+    model: str | None = None,
+    competition: str | None = None,
+):
+    """
+    Parameters
+    ----------
+    competition :
+        Competition name.
+
+    Auto R&D Evolving loop for models finetune.
+    You can continue running a session by using the command:
+    .. code-block:: bash
+        dotenv run -- python rdagent/app/finetune/data_science/loop.py --competition aerial-cactus-identification
+    """
+    if not competition:
+        raise Exception("Please specify competition name.")
+
+    model_folder = Path(DS_RD_SETTING.local_data_path) / competition / "prev_model"
+    if not model_folder.exists():
+        raise Exception(f"Please put the model path to {model_folder}.")
+    update_settings(competition)
+    rd_loop: DataScienceRDLoop = DataScienceRDLoop(DS_RD_SETTING)
+    asyncio.run(rd_loop.run())
+
+
+if __name__ == "__main__":
+    fire.Fire(main)

rdagent/app/finetune/data_science/scen.py

@@ -0,0 +1,20 @@
+from pathlib import Path
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.core.scenario import Scenario
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.data_science.scen import DataScienceScen
+from rdagent.scenarios.data_science.scen.utils import describe_data_folder_v2
+from rdagent.utils.agent.tpl import T
+
+
+class DSFinetuneScen(DataScienceScen):
+    """DSFinetuneScen Scenario"""
+
+    def _get_data_folder_description(self) -> str:
+        folder_desc = describe_data_folder_v2(
+            Path(DS_RD_SETTING.local_data_path) / self.competition,
+            show_nan_columns=DS_RD_SETTING.show_nan_columns,
+            max_length=20000,  # more context for model script
+        )
+        return folder_desc

rdagent/app/finetune/data_science/tpl/components/coder/data_science/pipeline/prompts.yaml

@@ -0,0 +1,4 @@
+pipeline_coder:
+  system: |-
+    {% include "rdagent.components.coder.data_science.pipeline.prompts:pipeline_coder.system" %}
+    NOTE: Ensure that base model form `{% include "scenarios.data_science.share:scen.input_path" %}prev_model` is correctly loaded, you are supposed to finetune the base model. 

rdagent/app/finetune/data_science/tpl/scenarios/data_science/proposal/exp_gen/prompts_v2.yaml

@@ -0,0 +1,6 @@
+task_gen:
+  system: |-
+    {% include "rdagent.scenarios.data_science.proposal.exp_gen.prompts_v2:task_gen.system" %}
+    NOTE: You MUST load base model form `{% include "scenarios.data_science.share:scen.input_path" %}prev_model`. Your main goal is to finetune it. 
+
+  

rdagent/app/finetune/llm/README.md

@@ -0,0 +1,256 @@
+# LLM Fine-tuning (FT) 场景运行指南
+
+本文档介绍如何运行 RD-Agent 的 LLM Fine-tuning 场景。
+
+## 简介
+
+FT 场景用于自动化优化大语言模型在特定 benchmark 上的表现。系统会自动：
+1. 生成数据处理和训练代码
+2. 执行模型微调
+3. 在目标 benchmark 上评估模型性能
+4. 根据反馈迭代改进
+
+## 支持的 Benchmark
+
+| 类别 | Benchmark | 数据集 | 描述 |
+|------|-----------|--------|------|
+| Math | `aime24`, `aime25` | `deepscaler` | AIME 数学竞赛 |
+| Patent | `panorama_par4pc` | `panorama-par4pc` | 专利现有技术检索 |
+| Patent | `panorama_pi4pc` | `panorama-pi4pc` | 专利段落识别 |
+| Patent | `panorama_noc4pc` | `panorama-noc4pc` | 专利新颖性分类 |
+| Chemistry | `chemcotbench_mol_und` | `chemcot-mol_und` | 分子理解 |
+| Chemistry | `chemcotbench_mol_edit` | `chemcot-mol_edit` | 分子编辑 |
+| Chemistry | `chemcotbench_mol_opt` | `chemcot-mol_opt` | 分子优化 |
+| Chemistry | `chemcotbench_reaction` | `chemcot-rxn` | 化学反应预测 |
+
+> 数据集配置位于 `rdagent/scenarios/finetune/datasets/__init__.py` 的 `DATASETS` 字典中。
+
+>运行时agent会查看所有数据集，根据target benchmark和scenario选出与之相关的。
+
+## 环境配置
+
+### 1. 运行环境
+
+确保已安装 `rdagent` 主运行环境，其他需要的运行环境会自动创建
+
+> 在 `.env` 配置文件中通过设置  `FT_Coder_CoSTEER_env_type = conda/docker` 来配置
+
+### 2. .env 配置文件
+
+在项目根目录创建 `.env` 文件，参考以下模板：
+
+```bash
+# ========== API Configuration ==========
+BACKEND=rdagent.oai.backend.LiteLLMAPIBackend
+CHAT_MODEL=gpt-5.2
+CHAT_TEMPERATURE=1
+CHAT_STREAM=True
+OPENAI_API_KEY=sk-xxx
+OPENAI_API_BASE=http://your-api-endpoint
+
+EMBEDDING_MODEL=text-embedding-ada-002
+EMBEDDING_USE_AZURE=True
+
+# ========== Global Configs ==========
+MAX_RETRY=12000
+RETRY_WAIT_SECONDS=5
+MULTI_PROC_N=16
+STEP_SEMAPHORE=1
+
+# ========== Cache Settings ==========
+DUMP_CHAT_CACHE=False
+USE_CHAT_CACHE=False
+DUMP_EMBEDDING_CACHE=True
+USE_EMBEDDING_CACHE=True
+LOG_LLM_CHAT_CONTENT=True
+
+CHAT_FREQUENCY_PENALTY=0.1
+CHAT_PRESENCE_PENALTY=0.0
+
+# ========== FT Scenario Specific ==========
+FT_FILE_PATH=/path/to/your/finetune/workspace
+
+# Environment type: docker or conda
+# Set to "conda" when Docker is unavailable
+FT_Coder_CoSTEER_env_type=conda
+
+# Docker settings (only used when env_type=docker)
+FT_DOCKER_ENABLE_CACHE=True
+FT_UPDATE_LLAMA_FACTORY=False
+
+# Data processing API concurrency (adjust based on target API capacity)
+FT_API_MAX_WORKERS=1000
+
+# Data processing Model
+FT_STRONG_MODELS='["gpt-5", "gpt-5.1"]'
+FT_WEAK_MODELS='["gpt-4o-mini"]'
+
+# Benchmark and target (can be overridden in script)
+FT_TARGET_BENCHMARK=aime25
+FT_USER_TARGET_SCENARIO="I need to enhance the model's performance on math reasoning tasks."
+
+# Timeout settings
+FT_DATA_PROCESSING_TIMEOUT=28800
+
+# Judge settings (optional)
+# FT_JUDGE_MODEL=gpt-5.1
+# FT_JUDGE_RETRY=10
+
+REASONING_THINK_RM=True
+
+# ========== Logging ==========
+LOG_FORMAT_CONSOLE="{time:YYYY-MM-DD HH:mm:ss.SSS} | {level: <8} | <cyan>{process}</cyan> | {name}:{function}:{line} - {message}"
+
+# ========== HuggingFace ==========
+HF_TOKEN=hf_xxx
+```
+
+## 运行方法
+
+### 基本命令
+
+```bash
+# 激活 conda 环境
+conda activate rdagent
+
+# 运行 FT 场景
+dotenv run -- python rdagent/app/finetune/llm/loop.py --base-model <MODEL>
+```
+
+### 命令行参数
+
+| 参数 | 说明 | 示例 |
+|------|------|------|
+| `--base-model` | 基础模型名称（必需，其他都可以不填） | `Qwen/Qwen2.5-7B-Instruct` |
+| `--benchmark` | 目标 benchmark | `aime25` |
+| `--benchmark-description` | Benchmark 描述 | - |
+| `--dataset` | 指定数据集 | - |
+| `--step-n` | 步数限制 | `10` |
+| `--loop-n` | 循环次数限制 | `5` |
+| `--timeout` | 总时间限制 | - |
+
+### 运行示例
+
+```bash
+# 在 AIME25 上微调 Qwen2.5-7B
+dotenv run -- python rdagent/app/finetune/llm/loop.py \
+    --base-model Qwen/Qwen2.5-7B-Instruct
+
+# 指定 GPU 运行
+CUDA_VISIBLE_DEVICES=0,1 dotenv run -- python rdagent/app/finetune/llm/loop.py \
+    --base-model Qwen/Qwen2.5-7B-Instruct
+
+# 限制循环次数
+dotenv run -- python rdagent/app/finetune/llm/loop.py \
+    --base-model Qwen/Qwen2.5-7B-Instruct \
+    --loop-n 3
+```
+
+### 多任务并行运行
+
+创建 `tasks.json` 配置文件：
+```json
+{
+  "tasks": [
+    {"model": "Qwen/Qwen2.5-7B-Instruct", "benchmark": "aime25", "gpus": "0,1"},
+    {"model": "Qwen/Qwen2.5-7B-Instruct", "benchmark": "gsm8k", "gpus": "2,3"}
+  ]
+}
+```
+
+使用 `run_ft_deploy.sh` 脚本运行：
+```bash
+./run_ft_deploy.sh tasks.json           # 正常运行
+./run_ft_deploy.sh tasks.json --dry-run # 仅预览配置
+./run_ft_deploy.sh tasks.json --no-sync # 禁用 blob 同步
+```
+
+<details>
+<summary>run_ft_deploy.sh 脚本参考</summary>
+
+```bash
+#!/bin/bash
+# 多任务并行部署脚本（简化版）
+
+RDAGENT_DIR="$HOME/RD-Agent"
+ENV_TEMPLATE=".env.ft"
+STAGGER_DELAY=60
+
+cd "$RDAGENT_DIR"
+source ~/miniconda3/etc/profile.d/conda.sh
+conda activate rdagent
+
+CONFIG_FILE="${1:-tasks.json}"
+NUM_TASKS=$(jq '.tasks | length' "$CONFIG_FILE")
+
+for ((i=0; i<NUM_TASKS; i++)); do
+    model=$(jq -r ".tasks[$i].model" "$CONFIG_FILE")
+    benchmark=$(jq -r ".tasks[$i].benchmark" "$CONFIG_FILE")
+    gpus=$(jq -r ".tasks[$i].gpus" "$CONFIG_FILE")
+
+    # 更新 .env 中的 benchmark
+    cp "$ENV_TEMPLATE" .env
+    sed -i "s|^FT_TARGET_BENCHMARK=.*|FT_TARGET_BENCHMARK=$benchmark|" .env
+
+    CUDA_VISIBLE_DEVICES=$gpus \
+    dotenv run -- python rdagent/app/finetune/llm/loop.py --base-model "$model" &
+
+    # 首个任务等待环境创建，后续任务错开启动
+    [[ $i -eq 0 ]] && sleep 120 || sleep $STAGGER_DELAY
+done
+
+wait
+```
+
+</details>
+
+## Blob 日志同步
+
+使用 Azure Blob 在多台机器间同步日志文件。
+
+### 1. 生成 SAS Token
+
+```bash
+# 首先登录 Azure CLI
+az login
+
+# 生成 Token（默认有效期 7 天）
+bash rdagent/utils/blob/gen_token.sh
+
+# 或指定过期时间
+bash rdagent/utils/blob/gen_token.sh 2025-01-31T00:00Z
+```
+
+Token 会保存到 `git_ignore_folder/.az_sas_token`。
+
+### 2. 同步日志
+
+同步路径：`log/` ↔ `blob://epeastus/rdagent/FinetuneAgenticLLM/FT_qizheng/logs`
+
+```bash
+# 上传本地日志到 Blob
+bash rdagent/utils/blob/azsync.sh up
+
+# 从 Blob 下载日志到本地
+bash rdagent/utils/blob/azsync.sh down
+```
+
+> 如需修改远程路径，编辑 `rdagent/utils/blob/azsync.sh` 中的 `REMOTE_PATH` 变量。
+
+## 日志查看
+
+运行日志保存在 `log/` 目录下：
+
+```
+log/
+└── 2025-01-01_12-00-00-123456/
+    ├── Loop_0/
+    │   ├── direct_exp_gen/   # 假设生成
+    │   ├── coding/           # 代码生成
+    │   ├── running/          # 训练执行
+    │   └── feedback/         # 反馈总结
+    └── Loop_1/
+        └── ...
+```
+
+

rdagent/app/finetune/llm/conf.py

@@ -0,0 +1,125 @@
+from pathlib import Path
+
+from pydantic_settings import SettingsConfigDict
+
+from rdagent.core.conf import ExtendedBaseSettings
+
+
+class LLMFinetunePropSetting(ExtendedBaseSettings):
+    """LLM Fine-tune dedicated property settings.
+
+    - Adjust timeouts and template
+    - Use FT_ env prefix for overrides
+    """
+
+    model_config = SettingsConfigDict(env_prefix="FT_", protected_namespaces=())
+
+    # Main Components
+    scen: str = "rdagent.scenarios.finetune.scen.scenario.LLMFinetuneScen"
+    """Scenario class for LLM fine-tuning tasks."""
+
+    hypothesis_gen: str = "rdagent.scenarios.finetune.proposal.proposal.LLMFinetuneExpGen"
+    """Hypothesis generation class for LLM fine-tuning tasks."""
+
+    coder: str = "rdagent.components.coder.finetune.LLMFinetuneCoSTEER"
+    """Code generator.
+    Function: Generate LLM fine-tuning code based on experiment design.
+    """
+
+    runner: str = "rdagent.scenarios.finetune.train.runner.LLMFinetuneRunner"  # TODO
+    """Code runner.
+    Function: Execute LLM fine-tuning code in a Docker environment.
+    """
+
+    summarizer: str = "rdagent.scenarios.finetune.dev.feedback.FTExperiment2Feedback"
+    """Result summarizer - To be implemented.
+    Function: Analyze fine-tuning results and generate feedback, including performance metrics and error analysis.
+    """
+
+    # Timeouts (longer for LLM training, all for Docker container timeout)
+    full_timeout: int = 360000
+    """Full training timeout in seconds (default 100 hours, env: FT_FULL_TIMEOUT). Used in running stage for complete model training."""
+    data_processing_timeout: int = 3600
+    """Data processing script timeout in seconds (default 1 hour, env: FT_DATA_PROCESSING_TIMEOUT). Used for full data processing in running stage."""
+    debug_data_processing_timeout: int = 1200
+    """Debug data processing timeout in seconds (default 20 minutes, env: FT_DEBUG_DATA_PROCESSING_TIMEOUT). Used for --debug mode in coding stage."""
+    micro_batch_timeout: int = 1800
+    """Micro-batch test timeout in seconds (default 30 minutes, env: FT_MICRO_BATCH_TIMEOUT)."""
+
+    # Pipeline behavior
+    coder_on_whole_pipeline: bool = True
+    app_tpl: str = "scenarios/finetune"
+
+    # Benchmark evaluation (always enabled as part of evaluation pipeline)
+
+    benchmark_timeout: int = 0
+    """Benchmark evaluation timeout in seconds. 0 means no timeout."""
+
+    # Judge API configuration (for llmjudge benchmarks like AIME)
+    judge_model: str = "gpt-5.1"
+    """LLM judge model name for evaluation"""
+
+    judge_api_key: str | None = None
+    """API key for judge model (if None, will try to use from environment)"""
+
+    judge_api_base: str | None = None
+    """API base URL for judge model (if None, will use default)"""
+
+    judge_retry: int = 10
+    """Number of retries for LLM judge API calls (env: FT_JUDGE_RETRY)"""
+
+    benchmark_limit: int | None = None
+    """Limit number of samples for benchmark evaluation (None for full evaluation). Use for quick testing and debugging."""
+
+    benchmark_num_runs: int = 1
+    """Number of times to run each sample (for computing average or pass@k). Set >1 for multiple runs."""
+
+    benchmark_pass_k: list[int] | None = None
+    """Pass@k parameter list for code generation tasks (e.g., [1, 5, 10]). None to disable."""
+
+    # Data paths and processing
+    file_path: Path = Path.cwd() / "git_ignore_folder" / "finetune_files"
+    show_nan_columns: bool = False
+    sample_data_by_LLM: bool = True
+
+    # LLM-specific fields
+    user_target_scenario: str | None = None
+    target_benchmark: str | None = None
+    """Benchmark dataset to evaluate on. Supported: aime25, aime24, mmlu, gsm8k, math, etc."""
+    benchmark_description: str | None = None
+    base_model: str | None = None
+    dataset: str | None = None
+    upper_data_size_limit: int = 2000
+
+    # Data processing LLM models (for API calls in data processing scripts)
+    strong_models: list[str] = ["gpt-5", "gpt-5.1"]
+    """Strong models for complex tasks (CoT generation, reasoning) - supports list (env: FT_STRONG_MODELS)"""
+
+    weak_models: list[str] = ["gpt-4o-mini", "o4-mini", "gpt-5-mini"]
+    """Weak models for simple tasks (filtering, format conversion) - supports list (env: FT_WEAK_MODELS)"""
+
+    embedding_models: list[str] = ["text-embedding-3-small", "text-embedding-3-large"]
+
+    # Docker settings
+    docker_enable_cache: bool = False
+    """Enable Docker cache for training (set via FT_DOCKER_ENABLE_CACHE)"""
+
+    # data sample count
+    data_sample_count: int = 3
+
+    # API concurrency for data processing
+    api_max_workers: int = 1000
+    """Max concurrent workers for LLM API calls in data processing scripts (env: FT_API_MAX_WORKERS)"""
+
+    # Coder settings
+    coder_max_loop: int = 10
+
+    # CoT format settings
+    force_think_token: bool = False
+    """Force <think> token wrapping for CoT training data (env: FT_FORCE_THINK_TOKEN).
+    When True: Data must be wrapped in <think>...</think> format, benchmark uses extract-non-reasoning-content postprocessor.
+    When False: CoT reasoning required but format is flexible, no postprocessor needed."""
+
+
+# Global setting instance for LLM finetuning scenario
+FT_RD_SETTING = LLMFinetunePropSetting()

rdagent/app/finetune/llm/job/README.md

@@ -0,0 +1,131 @@
+# FT Job Runner
+
+批量并行运行多个 LLM 微调任务的脚本。
+
+## 快速开始
+
+```bash
+# 1. 准备环境配置
+cp .env.template .env
+# 编辑 .env，填入 API key 等配置
+
+# 2. 准备任务配置
+cp tasks.json.example tasks.json
+# 编辑 tasks.json，定义要运行的任务
+
+# 3. 运行
+./run_ft_job.sh
+```
+
+## 用法
+
+```bash
+./run_ft_job.sh [tasks.json]
+```
+
+| 参数 | 说明 |
+|------|------|
+| `tasks.json` | 任务配置文件路径（可选，默认使用同目录下的 `tasks.json`） |
+| `-h, --help` | 显示帮助信息 |
+
+### 示例
+
+```bash
+# 使用默认配置
+./run_ft_job.sh
+
+# 指定自定义配置文件
+./run_ft_job.sh /path/to/my_tasks.json
+```
+
+## 配置文件
+
+### tasks.json
+
+定义要并行运行的任务列表：
+
+```json
+{
+  "tasks": [
+    {
+      "model": "Qwen/Qwen3-8B",
+      "benchmark": "aime25",
+      "gpus": "0,1"
+    },
+    {
+      "model": "Qwen/Qwen3-8B",
+      "benchmark": "gsm8k",
+      "gpus": "2,3",
+      "scenario": "自定义优化目标"
+    }
+  ]
+}
+```
+
+| 字段 | 必填 | 默认值 | 说明 |
+|------|:----:|--------|------|
+| `model` | ✅ | - | HuggingFace 模型路径 |
+| `benchmark` | ✅ | - | 评估基准（如 `aime25`, `gsm8k`） |
+| `gpus` | ❌ | `"0"` | 使用的 GPU 编号 |
+| `scenario` | ❌ | `"Improve model performance on {benchmark}"` | 优化目标描述 |
+
+### .env
+
+环境配置文件，包含 API 密钥、模型设置等。从 `.env.template` 复制并修改：
+
+```bash
+cp .env.template .env
+```
+
+主要配置项：
+
+| 配置 | 说明 |
+|------|------|
+| `OPENAI_API_KEY` | OpenAI API 密钥 |
+| `OPENAI_API_BASE` | API 地址 |
+| `FT_Coder_CoSTEER_env_type` | 环境类型：`docker` 或 `conda` |
+| `HF_TOKEN` | HuggingFace Token |
+
+## 输出
+
+运行后会在 `log/` 目录下创建 job 文件夹：
+
+```
+log/2025-12-23/
+├── aime25_Qwen3-8B.log      # 任务日志
+├── gsm8k_Qwen3-8B.log
+└── aime25_Qwen3-8B/         # 任务 trace（Loop 数据）
+    ├── Loop_0/
+    └── ...
+```
+
+## 监控
+
+### 命令行
+
+```bash
+# 查看所有任务日志
+tail -f log/2025-12-23/*.log
+
+# 查看特定任务
+tail -f log/2025-12-23/aime25_Qwen3-8B.log
+```
+
+### Web UI
+
+```bash
+streamlit run rdagent/app/finetune/llm/ui/app.py
+```
+
+在 UI 中选择 Job Folder 为对应的日志目录即可查看运行状态。
+
+## 依赖
+
+- `jq`：JSON 解析工具
+- `conda` 环境：`rdagent`
+
+## 注意事项
+
+1. 任务启动间隔默认为 60 秒（`STAGGER_DELAY`），避免同时启动造成资源竞争
+2. 确保指定的 GPU 编号不冲突
+3. 如果同一天多次运行，会自动创建 `log/2025-12-23_1/`、`log/2025-12-23_2/` 等目录

rdagent/app/finetune/llm/job/run_ft_job.sh

@@ -0,0 +1,186 @@
+#!/bin/bash
+# Run multiple FT tasks in parallel under a single job directory
+#
+# Usage: ./run_ft_job.sh [tasks.json]
+#
+# Config format (tasks.json):
+# {
+#   "tasks": [
+#     {"model": "Qwen/Qwen3-8B", "benchmark": "aime25", "gpus": "0,1"},
+#     {"model": "Qwen/Qwen3-8B", "benchmark": "gsm8k", "gpus": "2,3"}
+#   ]
+# }
+
+set -e
+
+# ========== CONFIG ==========
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RDAGENT_DIR="$(cd "$SCRIPT_DIR/../../../../.." && pwd)"
+ENV_FILE="$SCRIPT_DIR/.env"
+SCENARIOS_FILE="$SCRIPT_DIR/scenarios.json"
+STAGGER_DELAY=60
+
+usage() {
+    echo "Usage: $0 [tasks.json]"
+    echo "Run multiple FT tasks under a single job directory."
+    echo "UI: streamlit run rdagent/app/finetune/llm/ui/app.py"
+    exit 0
+}
+
+# ========== PARSE ARGS ==========
+CONFIG_FILE=""
+
+for arg in "$@"; do
+    case $arg in
+        -h|--help) usage ;;
+        *) [[ -z "$CONFIG_FILE" ]] && CONFIG_FILE="$arg" ;;
+    esac
+done
+
+[[ -z "$CONFIG_FILE" ]] && CONFIG_FILE="$SCRIPT_DIR/tasks.json"
+[[ ! -f "$CONFIG_FILE" ]] && echo "Error: Config not found: $CONFIG_FILE" && exit 1
+
+# Check .env file
+if [[ ! -f "$ENV_FILE" ]]; then
+    echo "Error: .env not found at $ENV_FILE"
+    echo "Please create it from template: cp $SCRIPT_DIR/.env.template $ENV_FILE"
+    exit 1
+fi
+
+# Check jq
+command -v jq &>/dev/null || { echo "Error: jq required"; exit 1; }
+
+# ========== SETUP ==========
+# Get log and workspace base paths from environment or use defaults
+# Default to project-relative paths; can be overridden by environment variables
+FT_LOG_BASE="${FT_LOG_BASE:-$RDAGENT_DIR/log}"
+FT_WORKSPACE_BASE="${FT_WORKSPACE_BASE:-$RDAGENT_DIR/git_ignore_folder/RD-Agent_workspace}"
+
+JOB_ID=$(date +%Y-%m-%d_%H-%M)
+JOB_DIR="$FT_LOG_BASE/$JOB_ID"
+if [[ -d "$JOB_DIR" ]]; then
+    i=1; while [[ -d "${JOB_DIR}_$i" ]]; do ((i++)); done
+    JOB_ID="${JOB_ID}_$i"; JOB_DIR="${JOB_DIR}_$i"
+fi
+mkdir -p "$JOB_DIR"
+
+cd "$RDAGENT_DIR"
+
+NUM_TASKS=$(jq '.tasks | length' "$CONFIG_FILE")
+
+echo "=============================================="
+echo "FT Job: $JOB_ID"
+echo "=============================================="
+echo "Config:    $CONFIG_FILE"
+echo "Tasks:     $NUM_TASKS"
+echo "Log:       $JOB_DIR"
+echo "Workspace: $FT_WORKSPACE_BASE/$JOB_ID"
+echo ""
+
+# Setup tmux session
+TMUX_SESSION="rdagent"
+tmux kill-session -t "$TMUX_SESSION" 2>/dev/null || true
+tmux new-session -d -s "$TMUX_SESSION" -n "main"
+echo "Tmux session created: $TMUX_SESSION"
+echo ""
+
+for ((i=0; i<NUM_TASKS; i++)); do
+    model=$(jq -r ".tasks[$i].model" "$CONFIG_FILE")
+    benchmark=$(jq -r ".tasks[$i].benchmark" "$CONFIG_FILE")
+    gpus=$(jq -r ".tasks[$i].gpus // \"0\"" "$CONFIG_FILE")
+    port=$(jq -r ".tasks[$i].port // empty" "$CONFIG_FILE")
+    task_timeout=$(jq -r ".tasks[$i].timeout // \"12h\"" "$CONFIG_FILE")
+
+    # Load benchmark_description: tasks.json -> scenarios.json
+    benchmark_desc=$(jq -r ".tasks[$i].benchmark_description // empty" "$CONFIG_FILE")
+    if [[ -z "$benchmark_desc" ]]; then
+        benchmark_desc=$(jq -r ".[\"$benchmark\"].benchmark_description // empty" "$SCENARIOS_FILE")
+    fi
+    # Note: Special characters in benchmark_desc are handled by writing to env file
+    model_name=$(basename "$model")
+    task_name="${benchmark}_${model_name}"
+    trace_path="$JOB_DIR/$task_name"
+
+    port_info=""
+    [[ -n "$port" ]] && port_info=", port=$port"
+    echo "Task $i: $task_name (model=$model, benchmark=$benchmark, gpus=$gpus$port_info)"
+
+    # Run task in tmux window with script -c for output capture
+    task_workspace="$FT_WORKSPACE_BASE/$JOB_ID/$task_name"
+    mkdir -p "$task_workspace"
+    LOG_FILE="$JOB_DIR/${task_name}.log"
+
+    # Write task-specific env file (avoids command-line escaping issues with special chars)
+    TASK_ENV_FILE="$task_workspace/.task_env"
+    cat > "$TASK_ENV_FILE" << EOF
+CUDA_VISIBLE_DEVICES='$gpus'
+LOG_TRACE_PATH='$trace_path'
+WORKSPACE_PATH='$task_workspace'
+FT_TARGET_BENCHMARK='$benchmark'
+EOF
+    # Escape shell special characters for double-quoted string: \ " ` $
+    if [[ -n "$benchmark_desc" ]]; then
+        escaped_desc="$benchmark_desc"
+        escaped_desc="${escaped_desc//\\/\\\\}"  # \ -> \\
+        escaped_desc="${escaped_desc//\"/\\\"}"  # " -> \"
+        escaped_desc="${escaped_desc//\`/\\\`}"  # ` -> \`
+        escaped_desc="${escaped_desc//\$/\\\$}"  # $ -> \$
+        echo "FT_BENCHMARK_DESCRIPTION=\"$escaped_desc\"" >> "$TASK_ENV_FILE"
+    fi
+    [[ -n "$port" ]] && echo "OPENAI_API_BASE='http://localhost:$port'" >> "$TASK_ENV_FILE"
+
+    # Create tmux window for this task and get its full target (e.g., rdagent:1.0)
+    # Use "session:" format to ensure window is created in the correct session
+    WIN_TARGET=$(tmux new-window -t "$TMUX_SESSION:" -n "$benchmark" -P)
+
+    # Build the command with environment setup (env vars loaded from file)
+    timeout_arg=""
+    [[ -n "$task_timeout" ]] && timeout_arg="--timeout $task_timeout"
+
+    TASK_CMD="source ~/miniconda3/etc/profile.d/conda.sh && conda activate qz_rdagent"
+    TASK_CMD="$TASK_CMD && set -a && source '$ENV_FILE' && source '$TASK_ENV_FILE' && set +a"
+    TASK_CMD="$TASK_CMD && cd '$RDAGENT_DIR'"
+    TASK_CMD="$TASK_CMD && python rdagent/app/finetune/llm/loop.py --base-model '$model' $timeout_arg"
+
+    # Run with script -c to capture terminal output (using full target for reliability)
+    tmux send-keys -t "$WIN_TARGET" "script -q '$LOG_FILE' -c \"$TASK_CMD\"" Enter
+
+    echo "  Window:    $benchmark"
+    echo ""
+
+    # Stagger starts
+    if [[ $i -eq 0 ]]; then
+        # First task: wait for initialization
+        # Get FT_FILE_PATH from .env or use default
+        FT_FILE_PATH=$(grep -E "^FT_FILE_PATH=" "$ENV_FILE" | cut -d= -f2 | tr -d '"' || echo "")
+        [[ -z "$FT_FILE_PATH" ]] && FT_FILE_PATH="$RDAGENT_DIR/git_ignore_folder/finetune"
+        DATASET_INFO="$FT_FILE_PATH/datasets/dataset_info.json"
+
+        echo "  Waiting for scenario initialization (dataset_info.json)..."
+        while [[ ! -f "$DATASET_INFO" ]]; do
+            sleep 5
+        done
+        echo "  Scenario initialized!"
+
+        echo "  Waiting for llm_finetune conda env..."
+        while ! conda run -n llm_finetune python -c "import requests" 2>/dev/null; do
+            sleep 10
+        done
+
+        echo "  Waiting for opencompass conda env..."
+        while ! conda run -n opencompass python -c "import opencompass" 2>/dev/null; do
+            sleep 10
+        done
+        echo "  Environment ready!"
+    elif [[ $i -lt $((NUM_TASKS - 1)) ]]; then
+        sleep $STAGGER_DELAY
+    fi
+done
+
+echo "=============================================="
+echo "All tasks started in tmux session: $TMUX_SESSION"
+echo "  - Attach:  tmux attach -t $TMUX_SESSION"
+echo "  - List:    tmux list-windows -t $TMUX_SESSION"
+echo "  - Select:  tmux select-window -t $TMUX_SESSION:{window_name}"
+echo "Monitor: tail -f $JOB_DIR/*.log"
+echo "UI: streamlit run rdagent/app/finetune/llm/ui/app.py (Job Folder: $JOB_DIR)"

rdagent/app/finetune/llm/job/scenarios.json

@@ -0,0 +1,128 @@
+{
+  "_comment": "Benchmark scenarios for FT tasks. Used by run_ft_job.sh and UI.",
+
+  "aime24": {
+    "category": "math",
+    "scenario": "Improve the model's ability to solve advanced competition math problems through multi-step reasoning, including number theory, combinatorics, geometry, and algebraic manipulation, with answers expressed as integers from 0 to 999.",
+    "benchmark_description": "AIME 2024 (American Invitational Mathematics Examination) - Advanced high school math competition problems requiring creative problem-solving. Each answer is an integer 0-999. Topics include number theory, algebra, geometry, trigonometry, probability, and combinatorics. Problems require multi-step reasoning and often have elegant solutions. Expected Output Format: Put final answer within \\boxed{}, e.g., \\boxed{42}."
+  },
+  "aime25": {
+    "category": "math",
+    "scenario": "Improve the model's ability to solve advanced competition math problems through multi-step reasoning, including number theory, combinatorics, geometry, and algebraic manipulation, with answers expressed as integers from 0 to 999.",
+    "benchmark_description": "AIME 2025 (American Invitational Mathematics Examination) - Advanced high school math competition problems requiring creative problem-solving. Each answer is an integer 0-999. Topics include number theory, algebra, geometry, trigonometry, probability, and combinatorics. Problems require multi-step reasoning and often have elegant solutions. Expected Output Format: Put final answer within \\boxed{}, e.g., \\boxed{42}."
+  },
+  "panorama": {
+    "category": "patent",
+    "scenario": "Improve the model's patent examination capabilities including prior art retrieval, paragraph identification, and novelty/obviousness classification based on USPTO standards.",
+    "benchmark_description": "PANORAMA tests patent examination capabilities based on real USPTO Office Actions. Tasks include: retrieving relevant prior art patents, identifying specific paragraphs in prior art that relate to claims, and classifying claims as allowable, lacking novelty (102), or obvious (103). Requires understanding patent law, technical document analysis, and legal reasoning. Expected Output Format: Return JSON with task-specific format (see subtask descriptions)."
+  },
+  "panorama_par4pc": {
+    "category": "patent",
+    "scenario": "Improve the model's ability to retrieve relevant prior art patents given a patent claim, by understanding claim scope, identifying technical similarities, and ranking patents by relevance for rejection analysis.",
+    "benchmark_description": "PAR4PC (Prior Art Retrieval for Patent Claims) - Given a patent claim, retrieve the most relevant prior art patents from a candidate pool. Requires understanding claim scope, identifying technical similarities, and ranking patents by relevance for potential 35 USC 102/103 rejections. Expected Output Format: Return JSON: {\"answer\": \"A\"} for single patent or {\"answer\": [\"A\", \"C\"]} for multiple patents (codes A-H)."
+  },
+  "panorama_pi4pc": {
+    "category": "patent",
+    "scenario": "Improve the model's ability to identify specific paragraphs in prior art patents that are most relevant for evaluating a claim's novelty and obviousness through element-by-element analysis.",
+    "benchmark_description": "PI4PC (Paragraph Identification for Patent Claims) - Given a patent claim and cited prior art patent, identify specific paragraphs most relevant for evaluating novelty and obviousness. Requires detailed technical reading, element-by-element claim analysis, and understanding how prior art teachings map to claim limitations. Expected Output Format: Return JSON: {\"answer\": \"<paragraph_id>\"}."
+  },
+  "panorama_noc4pc": {
+    "category": "patent",
+    "scenario": "Improve the model's ability to classify patent claims as allowable, anticipated, or obvious by applying patent law standards to analyze claim limitations against prior art.",
+    "benchmark_description": "NOC4PC (Novelty/Obviousness Classification) - Classify patent claims as ALLOW (patentable), 102 (anticipated/lacks novelty), or 103 (obvious). Requires applying patent law standards: 102 when single reference discloses all elements, 103 when combination of references with motivation makes claim obvious to skilled artisan. Expected Output Format: Return JSON: {\"code\": \"ALLOW\"} or {\"code\": \"102\"} or {\"code\": \"103\"}."
+  },
+  "panorama_par4pc_cot": {
+    "category": "patent",
+    "scenario": "Improve the model's ability to retrieve relevant prior art patents while providing explicit chain-of-thought reasoning explaining which claim elements each patent teaches and how it supports a rejection.",
+    "benchmark_description": "PAR4PC with chain-of-thought - Retrieve relevant prior art while providing explicit reasoning. Explain why each retrieved patent is relevant: which claim elements it teaches, what technical problems it addresses, and how it could support a rejection. Expected Output Format: Provide reasoning first, then return JSON: {\"answer\": \"A\"} or {\"answer\": [\"A\", \"C\"]}."
+  },
+  "panorama_pi4pc_cot": {
+    "category": "patent",
+    "scenario": "Improve the model's ability to identify relevant prior art paragraphs while providing element-by-element mapping showing how specific paragraph teachings correspond to claim limitations.",
+    "benchmark_description": "PI4PC with chain-of-thought - Identify relevant prior art paragraphs while explaining the technical connections. Provide element-by-element mapping showing how specific paragraph teachings correspond to claim limitations. Expected Output Format: Provide reasoning first, then return JSON: {\"answer\": \"<paragraph_id>\"}."
+  },
+  "panorama_noc4pc_cot": {
+    "category": "patent",
+    "scenario": "Improve the model's ability to classify patent claims with examiner-style rationale, explaining how references anticipate limitations or how combinations with motivation render claims obvious.",
+    "benchmark_description": "NOC4PC with chain-of-thought - Classify claims with examiner-style rationale. For 102: explain how reference anticipates each limitation. For 103: identify references, explain motivation to combine, and show how combination renders claim obvious. Use proper USPTO citation format. Expected Output Format: Return JSON: {\"reason\": \"<Office Action analysis>\", \"code\": \"ALLOW\"|\"102\"|\"103\"}."
+  },
+
+  "chemcotbench": {
+    "category": "chemistry",
+    "scenario": "Improve the model's chemical reasoning capabilities on molecular structures including understanding molecular features, editing molecules, optimizing properties, and predicting reaction outcomes.",
+    "benchmark_description": "ChemCoTBench tests step-wise chemical reasoning on SMILES molecular structures. Tasks include molecule understanding (identify functional groups, ring systems), molecule editing (add/delete/substitute groups while maintaining validity), molecule optimization (modify for desired properties), and reaction prediction (products, mechanisms, conditions). Contains subtasks with different output requirements. Expected Output Format: Return JSON: {\"output\": \"<answer>\"} where answer format depends on subtask - SMILES string for molecular tasks, numeric count for counting tasks, or Yes/No for equivalence tasks."
+  },
+  "chemcotbench_mol_und": {
+    "category": "chemistry",
+    "scenario": "Improve the model's ability to analyze molecular structures and identify structural features including functional groups (hydroxyl, carboxyl, amine), ring systems (aromatic, aliphatic), and molecular scaffolds.",
+    "benchmark_description": "Molecule Understanding - Analyze SMILES strings for structural features. Subtasks: (1) fg_count/ring_count: return integer count, (2) equivalence/ring_system_scaffold: return Yes or No, (3) Murcko_scaffold: return SMILES string. Requires parsing SMILES notation and applying organic chemistry knowledge. Expected Output Format: Return JSON: {\"output\": \"<answer>\"} where answer is integer/Yes/No/SMILES depending on subtask."
+  },
+  "chemcotbench_mol_edit": {
+    "category": "chemistry",
+    "scenario": "Improve the model's ability to perform precise structural modifications on molecules (add, delete, substitute functional groups) while maintaining chemical validity and molecule integrity.",
+    "benchmark_description": "Molecule Editing - Perform structural modifications on SMILES. Subtasks: add (add functional group), delete (remove group), sub (substitute group). Output must be valid SMILES representing chemically feasible molecules. Expected Output Format: Return JSON: {\"output\": \"<valid SMILES>\"}. SMILES validity is verified using RDKit."
+  },
+  "chemcotbench_mol_opt": {
+    "category": "chemistry",
+    "scenario": "Improve the model's ability to modify molecular structures to achieve target properties such as improved solubility, drug-likeness, or binding affinity to specific biological targets.",
+    "benchmark_description": "Molecule Optimization - Modify structures to achieve target properties. Subtasks: drd/gsk/jnk (binding affinity to DRD2/GSK3β/JNK3 targets), logp (lipophilicity), qed (drug-likeness), solubility. Requires understanding structure-property relationships. Expected Output Format: Return JSON: {\"output\": \"<optimized SMILES>\"}."
+  },
+  "chemcotbench_reaction": {
+    "category": "chemistry",
+    "scenario": "Improve the model's ability to predict chemical reaction outcomes including forward synthesis, retrosynthesis, mechanism selection, and reaction conditions based on functional group transformations.",
+    "benchmark_description": "Reaction Prediction - Predict reaction outcomes. Subtasks: fs (forward synthesis: reactants→products), retro (retrosynthesis: products→reactants), rcr (reaction condition recommendation), nepp (named reaction prediction), mechsel (mechanism selection). Requires understanding reaction types and functional group transformations. Expected Output Format: Return JSON: {\"output\": \"<SMILES or text answer>\"}."
+  },
+
+  "tablebench_data_analysis": {
+    "category": "table_qa",
+    "scenario": "Improve the model's ability to analyze tabular data for complex questions including trend identification, correlation analysis, statistical computation, and data-driven forecasting.",
+    "benchmark_description": "Table Data Analysis - Analyze tabular data to answer complex questions. Subtask types with different evaluation: (1) CorrelationAnalysis/TrendForecasting/StatisticalAnalysis: numeric answers with ±10% relative error tolerance, (2) ImpactAnalysis: exact match required, (3) Other analysis types: evaluated using ROUGE-L. Requires reading tables accurately and applying analytical reasoning. Expected Output Format: End response with \"Final Answer: <value>\"."
+  },
+  "tablebench_fact_checking": {
+    "category": "table_qa",
+    "scenario": "Improve the model's ability to verify factual claims against tabular data through accurate data extraction, implicit relationship understanding, and multi-hop reasoning across table cells.",
+    "benchmark_description": "Table Fact Checking - Answer table-based factual questions accurately. Questions may ask for specific information (numbers, names, dates) or verification (Yes/No, True/False). Uses Exact Match evaluation. Expected Output Format: End response with \"Final Answer: <value>\" where value is the precise answer to the question."
+  },
+  "tablebench_numerical_reasoning": {
+    "category": "table_qa",
+    "scenario": "Improve the model's ability to perform mathematical operations on table data including arithmetic, aggregations (sum, average, count), comparisons, percentages, and multi-step calculations.",
+    "benchmark_description": "Table Numerical Reasoning - Perform mathematical operations on table data: arithmetic (sum, difference, product), aggregations (average, count, max/min), comparisons, percentages, and multi-step calculations. Requires accurate number extraction and correct mathematical computation. Expected Output Format: End response with \"Final Answer: <numeric value>\"."
+  },
+  "tablebench_visualization": {
+    "category": "table_qa",
+    "scenario": "Improve the model's ability to generate Python code that creates appropriate visualizations (bar, line, pie, scatter charts) from tabular data with correct chart type selection and data mapping.",
+    "benchmark_description": "Table Visualization - Generate Python code to create appropriate visualizations from tabular data: bar charts, line charts, pie charts, scatter plots. Select correct chart type for data, map columns correctly to axes, and produce executable matplotlib/pandas code. Expected Output Format: Return Python code in ```python code block using matplotlib/pandas. Code will be executed to verify correctness."
+  },
+  "tablebench_gen": {
+    "category": "table_qa",
+    "scenario": "Improve the model's overall table question answering capabilities across fact checking, numerical reasoning, data analysis, and visualization by understanding table structure and generating accurate answers.",
+    "benchmark_description": "TableBench General - Comprehensive table QA covering fact checking, numerical reasoning, data analysis, and visualization. Questions require understanding table structure, extracting relevant data, performing reasoning or computation, and generating accurate answers or code. Expected Output Format: End response with \"Final Answer: <answer>\"."
+  },
+
+  "FinanceIQ_gen": {
+    "category": "finance",
+    "scenario": "Improve the model's financial domain knowledge and reasoning capabilities across Chinese financial certification exams including CPA, banking, securities, fund, futures, insurance, tax, and actuarial qualifications through multiple-choice question answering.",
+    "benchmark_description": "FinanceIQ tests financial domain knowledge through multiple-choice questions (A/B/C/D). Covers 10 Chinese financial certification exams: CPA (注册会计师), banking qualification, securities qualification, fund qualification, futures qualification, insurance qualification (CICE), tax advisor, economist, financial planner, and actuary. Uses LLM Judge for evaluation with 5-shot in-context learning. Evaluation metric: accuracy."
+  },
+
+  "bioprobench_gen": {
+    "category": "biology",
+    "scenario": "Improve the model's ability to generate complete, detailed experimental protocol steps from research context, including specific reagent concentrations, temperatures, incubation times, and equipment settings.",
+    "benchmark_description": "Protocol Generation - Generate complete experimental protocol steps given research context and objectives. Output detailed, actionable instructions: specify reagent concentrations, temperatures, incubation times, equipment settings. Protocols must be scientifically valid and reproducible. Expected Output Format: Wrap protocol steps in [ANSWER_START]Step 1: ... Step 2: ...[ANSWER_END]. Evaluated using BLEU, ROUGE, and step matching metrics."
+  },
+  "bioprobench_ord": {
+    "category": "biology",
+    "scenario": "Improve the model's ability to arrange shuffled experimental steps in correct sequence. Output MUST be a valid Python list format: [ANSWER_START][0, 2, 1, 3][ANSWER_END]. Use brackets and commas, NOT space-separated indices.",
+    "benchmark_description": "Step Ordering - Arrange shuffled experimental procedure steps in correct logical and temporal sequence. Requires understanding procedural dependencies: which steps must precede others, timing constraints, and scientific logic of experimental workflows. CRITICAL OUTPUT FORMAT: Answer MUST be a valid Python list with brackets and commas, e.g., [ANSWER_START][2, 0, 1, 3][ANSWER_END]. NOT space-separated (0 2 1 3 is WRONG), NOT without brackets (0, 2, 1, 3 is WRONG). Evaluated using Exact Match and Kendall's Tau."
+  },
+  "bioprobench_err": {
+    "category": "biology",
+    "scenario": "Improve the model's ability to identify errors in biological protocol text. CRITICAL SEMANTICS: True = step is CORRECT (no errors), False = step HAS ERRORS. This matches the benchmark prompt: 'If you find anything wrong, answer False.' Output format: [ANSWER_START]True or False[ANSWER_END].",
+    "benchmark_description": "Error Correction - Identify errors in biological protocol text including incorrect temperatures, concentrations, reagents, or procedural mistakes. CRITICAL: The benchmark expects True if the protocol step is CORRECT (no errors), and False if it HAS ERRORS. This follows the prompt: 'If you find anything wrong, answer False.' Do NOT invert this logic. Expected Output Format: [ANSWER_START]True[ANSWER_END] for correct steps, [ANSWER_START]False[ANSWER_END] for erroneous steps."
+  },
+  "bioprobench_pqa": {
+    "category": "biology",
+    "scenario": "Improve the model's ability to extract specific factual information from experimental protocols including temperatures, concentrations, incubation times, reagent quantities, and procedural details.",
+    "benchmark_description": "Protocol QA - Extract specific factual information from experimental protocols: temperatures, concentrations, incubation times, reagent quantities, equipment specifications, and procedural details. Requires careful reading and accurate information extraction from technical text. Expected Output Format: Return [ANSWER_START]<answer text> & <confidence 0-100>%[ANSWER_END], e.g., [ANSWER_START]Option A & 95%[ANSWER_END]. Evaluated using accuracy and Brier Score."
+  }
+}

rdagent/app/finetune/llm/job/tasks.json.example

@@ -0,0 +1,20 @@
+{
+  "tasks": [
+    {
+      "model": "Qwen/Qwen3-8B",
+      "benchmark": "aime25",
+      "gpus": "0,1"
+    },
+    {
+      "model": "Qwen/Qwen3-8B",
+      "benchmark": "gsm8k",
+      "gpus": "2,3"
+    },
+    {
+      "model": "meta-llama/Llama-3-8B",
+      "benchmark": "aime25",
+      "gpus": "4,5",
+      "scenario": "Improve AIME 2025 math reasoning with custom approach"
+    }
+  ]
+}

rdagent/app/finetune/llm/loop.py

@@ -0,0 +1,100 @@
+"""
+LLM Fine-tuning Entry Point
+
+Standard RDLoop entry point for LLM fine-tuning, consistent with data science implementation.
+"""
+
+import asyncio
+from typing import Optional, cast
+
+import fire
+
+from rdagent.app.finetune.llm.conf import FT_RD_SETTING
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.finetune.loop import LLMFinetuneRDLoop
+
+
+def main(
+    path: Optional[str] = None,
+    checkout: bool = True,
+    user_target_scenario: Optional[str] = None,
+    benchmark: Optional[str] = None,
+    benchmark_description: Optional[str] = None,
+    dataset: Optional[str] = None,
+    base_model: Optional[str] = None,
+    upper_data_size_limit: Optional[int] = None,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    timeout: Optional[str] = None,
+):
+    """
+    LLM fine-tuning entry point
+
+    Parameters
+    ----------
+    path :
+        A path like `$LOG_PATH/__session__/1/0_propose`. This indicates that we restore the state after finishing step 0 in loop 1.
+    checkout :
+        Used to control the log session path. Boolean type, default is True.
+        - If True, the new loop will use the existing folder and clear logs for sessions after the one corresponding to the given path.
+        - If False, the new loop will use the existing folder but keep the logs for sessions after the one corresponding to the given path.
+    dataset : str
+        Dataset name for fine-tuning (e.g., 'shibing624/alpaca-zh')
+    base_model : str, optional
+        Model name for fine-tuning (e.g., 'Qwen/Qwen2.5-1.5B-Instruct').
+        If not provided, auto-selects optimal model based on hardware and dataset.
+    step_n : int, optional
+        Number of steps to run; if None, runs indefinitely until completion or error
+    loop_n : int, optional
+        Number of loops to run; if None, runs indefinitely until completion or error
+    timeout : str, optional
+        Maximum duration for the entire process
+
+    Examples:
+    .. code-block:: bash
+        dotenv run -- python rdagent/app/finetune/llm/loop.py --dataset shibing624/alpaca-zh --base-model Qwen/Qwen2.5-1.5B-Instruct
+        dotenv run -- python rdagent/app/finetune/llm/loop.py --dataset shibing624/alpaca-zh    # TODO: not enabled yet
+    """
+
+    if user_target_scenario:
+        FT_RD_SETTING.user_target_scenario = user_target_scenario
+    assert (
+        FT_RD_SETTING.user_target_scenario is None
+    ), "user_target_scenario is not yet supported, please specify via benchmark and benchmark_description"
+    if upper_data_size_limit:
+        FT_RD_SETTING.upper_data_size_limit = upper_data_size_limit
+        logger.info(f"Set upper_data_size_limit to {FT_RD_SETTING.upper_data_size_limit}")
+    if benchmark and benchmark_description:
+        FT_RD_SETTING.target_benchmark = benchmark
+        FT_RD_SETTING.benchmark_description = benchmark_description
+    assert FT_RD_SETTING.user_target_scenario or (
+        FT_RD_SETTING.target_benchmark and FT_RD_SETTING.benchmark_description
+    ), "Either user_target_scenario or target_benchmark must be specified for LLM fine-tuning."
+
+    # Update configuration with provided parameters
+    if dataset:
+        FT_RD_SETTING.dataset = dataset
+    if base_model:
+        FT_RD_SETTING.base_model = base_model
+
+    # Create and run LLM fine-tuning loop
+    data_set_target = FT_RD_SETTING.dataset if FT_RD_SETTING.dataset else "auto generated dataset"
+    model_target = FT_RD_SETTING.base_model if FT_RD_SETTING.base_model else "auto selected model"
+
+    # Temporary assertion until auto-selection is implemented
+    assert (
+        FT_RD_SETTING.base_model is not None
+    ), "Base model auto selection not yet supported, please specify via --base-model"
+
+    logger.info(f"Starting LLM fine-tuning on dataset='{data_set_target}' with model='{model_target}'")
+
+    if path is None:
+        loop = LLMFinetuneRDLoop(FT_RD_SETTING)
+    else:
+        loop = cast(LLMFinetuneRDLoop, LLMFinetuneRDLoop.load(str(path), checkout=checkout))
+
+    asyncio.run(loop.run(step_n=step_n, loop_n=loop_n, all_duration=timeout))
+
+
+if __name__ == "__main__":
+    fire.Fire(main)

rdagent/app/finetune/llm/ui/__init__.py

@@ -0,0 +1 @@
+# FT (Fine-tune) scenario UI

rdagent/app/finetune/llm/ui/app.py

@@ -0,0 +1,207 @@
+"""
+FT (Fine-tune) Timeline Viewer
+Hierarchical view: Session > Loop > Stage > EvoLoop > Events
+
+Run:
+    streamlit run rdagent/app/finetune/llm/ui/app.py
+"""
+
+import os
+from pathlib import Path
+
+import streamlit as st
+from streamlit import session_state as state
+
+from rdagent.app.finetune.llm.ui.benchmarks import get_core_metric_score
+from rdagent.app.finetune.llm.ui.components import render_session, render_summary
+from rdagent.app.finetune.llm.ui.config import ALWAYS_VISIBLE_TYPES, OPTIONAL_TYPES
+from rdagent.app.finetune.llm.ui.data_loader import (
+    get_summary,
+    get_valid_sessions,
+    load_ft_session,
+)
+from rdagent.app.finetune.llm.ui.ft_summary import render_job_summary
+
+DEFAULT_LOG_BASE = "log/"
+
+
+def get_job_options(base_path: Path) -> list[str]:
+    """
+    Scan directory and return job options list.
+    - "." means standalone tasks in root directory
+    - Others are job directory names
+    """
+    options = []
+    has_root_tasks = False
+    job_dirs = []
+
+    if not base_path.exists():
+        return options
+
+    for d in base_path.iterdir():
+        if not d.is_dir():
+            continue
+        # Check if standalone task (has __session__ directly)
+        if (d / "__session__").exists():
+            has_root_tasks = True
+        # Check if job directory (subdirs have __session__)
+        else:
+            try:
+                if any((sub / "__session__").exists() for sub in d.iterdir() if sub.is_dir()):
+                    job_dirs.append(d.name)
+            except PermissionError:
+                pass
+
+    # Sort job dirs by name descending (newest first, since names are date-based)
+    job_dirs.sort(reverse=True)
+
+    # Add job dirs first, then root tasks at the end
+    options.extend(job_dirs)
+    if has_root_tasks:
+        options.append(". (Current)")
+
+    return options
+
+
+def main():
+    st.set_page_config(layout="wide", page_title="FT Timeline", page_icon="🔬")
+
+    # ========== Sidebar ==========
+    with st.sidebar:
+        # View mode selection
+        view_mode = st.radio("View Mode", ["Job Summary", "Single Task"], horizontal=True)
+
+        st.divider()
+
+        default_log = os.environ.get("FT_LOG_PATH", DEFAULT_LOG_BASE)
+        job_folder = default_log  # Initialize for both modes
+        selected_types = ALWAYS_VISIBLE_TYPES.copy()  # Initialize for both modes
+        is_root_job = False  # Track if viewing root tasks
+
+        if view_mode == "Job Summary":
+            # Job Summary mode
+            st.header("Job")
+            base_folder = st.text_input("Base Folder", value=default_log, key="base_folder_input")
+            base_path = Path(base_folder)
+
+            job_options = get_job_options(base_path)
+            if job_options:
+                selected_job = st.selectbox("Select Job", job_options, key="job_select")
+                if selected_job.startswith("."):
+                    job_folder = base_folder
+                    is_root_job = True
+                else:
+                    job_folder = str(base_path / selected_job)
+                # Save to session_state for Single Task mode
+                state.selected_job_folder = job_folder
+            else:
+                st.warning("No jobs found in this directory")
+                job_folder = base_folder
+
+            if st.button("Refresh", type="primary", key="refresh_job"):
+                st.rerun()
+        else:
+            # Single Task mode
+            st.header("Session")
+            # Use job_folder from Job Summary mode if available
+            default_path = getattr(state, "selected_job_folder", default_log)
+            log_folder = st.text_input("Log Folder", value=default_path)
+            log_path = Path(log_folder)
+
+            sessions = get_valid_sessions(log_path)
+            if not sessions:
+                st.warning("No valid sessions found")
+                return
+
+            selected_session = st.selectbox("Session", sessions)
+
+            if st.button("Load", type="primary") or "session" not in state:
+                with st.spinner("Loading..."):
+                    state.session = load_ft_session(log_path / selected_session)
+                    state.session_name = selected_session
+
+            st.divider()
+
+            # Optional type toggles
+            st.subheader("Show More")
+            selected_types = ALWAYS_VISIBLE_TYPES.copy()
+            for event_type, (label, default) in OPTIONAL_TYPES.items():
+                if st.toggle(label, value=default, key=f"toggle_{event_type}"):
+                    selected_types.append(event_type)
+
+            st.divider()
+
+            # Display options
+            st.subheader("Display Options")
+            state.render_markdown = st.toggle("Render Prompts", value=False, key="render_markdown_toggle")
+
+            st.divider()
+
+            # Summary in sidebar
+            if "session" in state:
+                summary = get_summary(state.session)
+                st.subheader("Summary")
+                st.metric("Loops", summary.get("loop_count", 0))
+                st.metric("LLM Calls", summary.get("llm_call_count", 0))
+                success = summary.get("docker_success", 0)
+                fail = summary.get("docker_fail", 0)
+                st.metric("Docker", f"{success}✓ / {fail}✗")
+
+    # ========== Main Content ==========
+    if view_mode == "Job Summary":
+        st.title("📊 FT Job Summary")
+        job_path = Path(job_folder)
+        if job_path.exists():
+            render_job_summary(job_path, is_root=is_root_job)
+        else:
+            st.warning(f"Job folder not found: {job_folder}")
+        return
+
+    # Single Task mode
+    st.title("🔬 FT Timeline Viewer")
+
+    if "session" not in state:
+        st.info("Select a session and click **Load** to view")
+        return
+
+    session = state.session
+    summary = get_summary(session)
+
+    # Global info header (Base Model, Datasets, Benchmark) - compact style
+    scenario_event = next((e for e in session.init_events if e.type == "scenario"), None)
+    dataset_event = next((e for e in session.init_events if e.type == "dataset_selection"), None)
+
+    if scenario_event or dataset_event:
+        if scenario_event and hasattr(scenario_event.content, "base_model"):
+            st.markdown(f"🧠 **Model:** `{scenario_event.content.base_model}`")
+        if dataset_event:
+            selected = (
+                dataset_event.content.get("selected_datasets", []) if isinstance(dataset_event.content, dict) else []
+            )
+            if selected:
+                st.markdown(f"📂 **Datasets:** `{', '.join(selected)}`")
+        if scenario_event and hasattr(scenario_event.content, "target_benchmark"):
+            st.markdown(f"🎯 **Benchmark:** `{scenario_event.content.target_benchmark}`")
+        # Display baseline benchmark score
+        if scenario_event and hasattr(scenario_event.content, "baseline_benchmark_score"):
+            baseline = scenario_event.content.baseline_benchmark_score
+            if baseline and isinstance(baseline, dict):
+                benchmark_name = getattr(scenario_event.content, "target_benchmark", "")
+                accuracy_summary = baseline.get("accuracy_summary", {})
+                if accuracy_summary:
+                    result = get_core_metric_score(benchmark_name, accuracy_summary)
+                    if result:
+                        metric_name, score, _ = result
+                        st.markdown(f"📊 **Baseline:** `{metric_name} = {score:.1f}`")
+
+    # Summary bar
+    render_summary(summary)
+
+    st.divider()
+
+    # Hierarchical view
+    render_session(session, selected_types)
+
+
+if __name__ == "__main__":
+    main()

rdagent/app/finetune/llm/ui/benchmarks/__init__.py

@@ -0,0 +1,70 @@
+"""Benchmark processors for core metric extraction.
+
+Each benchmark has its own processor that knows how to extract
+the core metric name and value from accuracy_summary data.
+"""
+
+from .bioprobench import BioProBenchProcessor
+from .chemcotbench import ChemCotBenchProcessor
+from .financeiq import FinanceIQProcessor
+from .panorama import PanoramaProcessor
+from .tablebench import TableBenchProcessor
+
+PROCESSORS = [
+    FinanceIQProcessor,
+    PanoramaProcessor,
+    ChemCotBenchProcessor,
+    TableBenchProcessor,
+    BioProBenchProcessor,
+]
+
+
+def get_core_metric_score(benchmark_name: str, accuracy_summary: dict) -> tuple[str, float, bool] | None:
+    """Get core metric name, score, and direction for a benchmark.
+
+    Args:
+        benchmark_name: The benchmark name (e.g., "FinanceIQ", "panorama_par4pc")
+        accuracy_summary: {dataset_name: {metric: value, ...}, ...}
+
+    Returns:
+        (metric_name, value, higher_is_better) or None
+        - metric_name: includes "(average)" suffix if multiple datasets are averaged
+        - value: the score
+        - higher_is_better: True if higher values are better (use ↑), False otherwise (use ↓)
+    """
+    for processor in PROCESSORS:
+        if processor.match(benchmark_name):
+            return processor.get_core_metric(accuracy_summary)
+
+    # Default fallback: use first numeric value with "accuracy" label
+    scores = []
+    for ds, metrics in accuracy_summary.items():
+        if not isinstance(metrics, dict):
+            continue
+        if "accuracy" in metrics:
+            scores.append(float(metrics["accuracy"]))
+        else:
+            for v in metrics.values():
+                if isinstance(v, (int, float)):
+                    scores.append(float(v))
+                    break
+
+    if not scores:
+        return None
+
+    avg = sum(scores) / len(scores)
+    if len(scores) == 1:
+        return ("accuracy", avg, True)  # higher is better
+    else:
+        return ("accuracy (average)", avg, True)  # higher is better
+
+
+__all__ = [
+    "get_core_metric_score",
+    "PROCESSORS",
+    "FinanceIQProcessor",
+    "PanoramaProcessor",
+    "ChemCotBenchProcessor",
+    "TableBenchProcessor",
+    "BioProBenchProcessor",
+]

rdagent/app/finetune/llm/ui/benchmarks/base.py

@@ -0,0 +1,53 @@
+"""Base class for benchmark core metric extraction."""
+
+from abc import ABC, abstractmethod
+
+
+class BenchmarkProcessor(ABC):
+    """Base class for benchmark core metric extraction."""
+
+    # Metrics where higher values are better (default assumption)
+    # Override in subclass if needed
+    HIGHER_IS_BETTER: set[str] = {
+        "accuracy",
+        "exact_match",
+        "f1",
+        "f1_score",
+        "macro_f1",
+        "correct_rate",
+        "success_rate",
+        "gold_hit_rate",
+        "score",
+        "scaffold_hard",
+        "kendall_tau",
+        "ROUGE-L",
+    }
+
+    @classmethod
+    @abstractmethod
+    def match(cls, benchmark_name: str) -> bool:
+        """Check if this processor handles the given benchmark."""
+        pass
+
+    @classmethod
+    @abstractmethod
+    def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
+        """Extract core metric name, value, and direction from accuracy_summary.
+
+        Args:
+            accuracy_summary: {dataset_name: {metric: value, ...}, ...}
+
+        Returns:
+            (metric_name, value, higher_is_better) or None
+            - metric_name: includes "(average)" suffix if multiple datasets
+            - value: the score
+            - higher_is_better: True if higher values are better, False otherwise
+        """
+        pass
+
+    @classmethod
+    def is_higher_better(cls, metric_name: str) -> bool:
+        """Check if higher values are better for this metric."""
+        # Remove (average) suffix for checking
+        base_metric = metric_name.replace(" (average)", "").strip()
+        return base_metric.lower() in {m.lower() for m in cls.HIGHER_IS_BETTER}

rdagent/app/finetune/llm/ui/benchmarks/bioprobench.py

@@ -0,0 +1,60 @@
+"""BioProBench benchmark processor."""
+
+from .base import BenchmarkProcessor
+
+
+class BioProBenchProcessor(BenchmarkProcessor):
+    """BioProBench: Biology protocol benchmark with different task types."""
+
+    CORE_METRICS = {
+        "pqa": "accuracy",
+        "ord": "kendall_tau",
+        "err": "f1",
+        "gen": "ROUGE-L",
+    }
+
+    @classmethod
+    def match(cls, benchmark_name: str) -> bool:
+        return "bioprobench" in benchmark_name.lower()
+
+    @classmethod
+    def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
+        scores = []
+        metrics_used = []
+
+        for ds, metrics in accuracy_summary.items():
+            if not isinstance(metrics, dict):
+                continue
+            ds_lower = ds.lower()
+            # Find matching core metric
+            core_metric = "accuracy"  # fallback
+            for pattern, metric in cls.CORE_METRICS.items():
+                if pattern in ds_lower:
+                    core_metric = metric
+                    break
+
+            if core_metric in metrics:
+                scores.append(float(metrics[core_metric]))
+                metrics_used.append(core_metric)
+            elif core_metric.lower() in [k.lower() for k in metrics.keys()]:
+                # Case-insensitive fallback for metrics like "ROUGE-L"
+                for k, v in metrics.items():
+                    if k.lower() == core_metric.lower():
+                        scores.append(float(v))
+                        metrics_used.append(core_metric)
+                        break
+
+        if not scores:
+            return None
+
+        avg = sum(scores) / len(scores)
+        unique = list(set(metrics_used))
+
+        if len(scores) == 1:
+            metric_name = unique[0]
+        elif len(unique) == 1:
+            metric_name = f"{unique[0]} (average)"
+        else:
+            metric_name = "mixed (average)"
+
+        return (metric_name, avg, cls.is_higher_better(metric_name))

rdagent/app/finetune/llm/ui/benchmarks/chemcotbench.py

@@ -0,0 +1,105 @@
+"""ChemCotBench benchmark processor."""
+
+from .base import BenchmarkProcessor
+
+
+class ChemCotBenchProcessor(BenchmarkProcessor):
+    """ChemCotBench: Chemistry reasoning with various subtasks.
+
+    All metrics are 0-100 percentages, enabling unified averaging within each subset.
+    """
+
+    # Define core metric field names for each task
+    CORE_METRICS = {
+        # Molecular understanding
+        "mol_und_fg_count": "accuracy",
+        "mol_und_ring_count": "accuracy",
+        "mol_und_murcko_scaffold": "scaffold_hard",  # Exact match rate (0-100%)
+        "mol_und_ring_system_scaffold": "score",  # "Yes" ratio (0-100%)
+        "mol_und_equivalence": "accuracy",
+        # Molecular editing
+        "mol_edit_add": "correct_rate",
+        "mol_edit_delete": "correct_rate",
+        "mol_edit_sub": "correct_rate",
+        # Molecular optimization (prefix match)
+        "mol_opt_": "success_rate",
+        # Reaction tasks - unified to exact_match
+        "reaction_fs": "exact_match",
+        "reaction_retro": "exact_match",
+        "reaction_nepp": "exact_match",
+        "reaction_rcr": "exact_match",
+        "reaction_mechsel": "exact_match",  # Will fallback to accuracy if exact_match not found
+    }
+
+    # Metric groups: unified display names for each subset
+    METRIC_GROUPS = {
+        "mol_und": "accuracy",  # mol_und subset displays as accuracy
+        "mol_edit": "correct_rate",
+        "mol_opt": "success_rate",
+        "reaction": "exact_match",  # reaction subset displays as exact_match
+    }
+
+    @classmethod
+    def match(cls, benchmark_name: str) -> bool:
+        return "chemcot" in benchmark_name.lower()
+
+    @classmethod
+    def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
+        scores = []
+        group_detected = None
+
+        for ds, metrics in accuracy_summary.items():
+            if not isinstance(metrics, dict):
+                continue
+            ds_lower = ds.lower()
+
+            # Detect subset type
+            for group in cls.METRIC_GROUPS:
+                if group in ds_lower:
+                    group_detected = group
+                    break
+
+            # Find matching core metric
+            core_metric = "accuracy"  # fallback
+            for pattern, metric in cls.CORE_METRICS.items():
+                # Prefix match for patterns ending with _
+                if pattern.endswith("_"):
+                    if pattern in ds_lower:
+                        core_metric = metric
+                        break
+                else:
+                    if pattern in ds_lower:
+                        core_metric = metric
+                        break
+
+            # Try to get metric value with fallback support
+            value = None
+            if core_metric in metrics:
+                value = float(metrics[core_metric])
+            elif core_metric == "exact_match" and "accuracy" in metrics:
+                # reaction_mechsel fallback: exact_match -> accuracy
+                value = float(metrics["accuracy"])
+
+            if value is not None:
+                scores.append(value)
+
+        if not scores:
+            return None
+
+        avg = sum(scores) / len(scores)
+
+        # Use unified metric name for the detected subset
+        if group_detected and group_detected in cls.METRIC_GROUPS:
+            unified_name = cls.METRIC_GROUPS[group_detected]
+            if len(scores) == 1:
+                metric_name = unified_name
+            else:
+                metric_name = f"{unified_name} (average)"
+        else:
+            # Fallback for unknown subsets
+            if len(scores) == 1:
+                metric_name = "accuracy"
+            else:
+                metric_name = "accuracy (average)"
+
+        return (metric_name, avg, cls.is_higher_better(metric_name))

rdagent/app/finetune/llm/ui/benchmarks/financeiq.py

@@ -0,0 +1,29 @@
+"""FinanceIQ benchmark processor."""
+
+from .base import BenchmarkProcessor
+
+
+class FinanceIQProcessor(BenchmarkProcessor):
+    """FinanceIQ: 10 exam subjects, all use accuracy."""
+
+    @classmethod
+    def match(cls, benchmark_name: str) -> bool:
+        return "financeiq" in benchmark_name.lower()
+
+    @classmethod
+    def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
+        scores = []
+        for ds, metrics in accuracy_summary.items():
+            if not isinstance(metrics, dict):
+                continue
+            if "accuracy" in metrics:
+                scores.append(float(metrics["accuracy"]))
+
+        if not scores:
+            return None
+
+        avg = sum(scores) / len(scores)
+        if len(scores) == 1:
+            return ("accuracy", avg, True)  # higher is better
+        else:
+            return ("accuracy (average)", avg, True)  # higher is better

rdagent/app/finetune/llm/ui/benchmarks/panorama.py

@@ -0,0 +1,52 @@
+"""Panorama benchmark processor."""
+
+from .base import BenchmarkProcessor
+
+
+class PanoramaProcessor(BenchmarkProcessor):
+    """Panorama: Different sub-datasets use different metrics."""
+
+    CORE_METRICS = {
+        "par4pc": "macro_f1",
+        "pi4pc": "gold_hit_rate",
+        "noc4pc": "macro_f1",
+    }
+
+    @classmethod
+    def match(cls, benchmark_name: str) -> bool:
+        return "panorama" in benchmark_name.lower()
+
+    @classmethod
+    def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
+        scores = []
+        metrics_used = []
+
+        for ds, metrics in accuracy_summary.items():
+            if not isinstance(metrics, dict):
+                continue
+            ds_lower = ds.lower()
+            # Find matching core metric
+            core_metric = "accuracy"  # fallback
+            for pattern, metric in cls.CORE_METRICS.items():
+                if pattern in ds_lower:
+                    core_metric = metric
+                    break
+
+            if core_metric in metrics:
+                scores.append(float(metrics[core_metric]))
+                metrics_used.append(core_metric)
+
+        if not scores:
+            return None
+
+        avg = sum(scores) / len(scores)
+        unique = list(set(metrics_used))
+
+        if len(scores) == 1:
+            metric_name = unique[0]
+        elif len(unique) == 1:
+            metric_name = f"{unique[0]} (average)"
+        else:
+            metric_name = "mixed (average)"
+
+        return (metric_name, avg, cls.is_higher_better(metric_name))

rdagent/app/finetune/llm/ui/benchmarks/tablebench.py

@@ -0,0 +1,60 @@
+"""TableBench benchmark processor."""
+
+from .base import BenchmarkProcessor
+
+
+class TableBenchProcessor(BenchmarkProcessor):
+    """TableBench: Table QA with different subtasks."""
+
+    CORE_METRICS = {
+        "fact": "accuracy",
+        "numerical": "accuracy",
+        "analysis": "accuracy",
+        "visualization": "Pass@1",  # TableBench visualization uses Pass@1 as core metric
+    }
+
+    # TableBench-specific metrics where higher is better
+    HIGHER_IS_BETTER = BenchmarkProcessor.HIGHER_IS_BETTER | {
+        "Pass@1",
+        "ECR@1",
+        "Parse@1",
+    }
+
+    @classmethod
+    def match(cls, benchmark_name: str) -> bool:
+        return "tablebench" in benchmark_name.lower()
+
+    @classmethod
+    def get_core_metric(cls, accuracy_summary: dict) -> tuple[str, float, bool] | None:
+        scores = []
+        metrics_used = []
+
+        for ds, metrics in accuracy_summary.items():
+            if not isinstance(metrics, dict):
+                continue
+            ds_lower = ds.lower()
+            # Find matching core metric
+            core_metric = "accuracy"  # fallback
+            for pattern, metric in cls.CORE_METRICS.items():
+                if pattern in ds_lower:
+                    core_metric = metric
+                    break
+
+            if core_metric in metrics:
+                scores.append(float(metrics[core_metric]))
+                metrics_used.append(core_metric)
+
+        if not scores:
+            return None
+
+        avg = sum(scores) / len(scores)
+        unique = list(set(metrics_used))
+
+        if len(scores) == 1:
+            metric_name = unique[0]
+        elif len(unique) == 1:
+            metric_name = f"{unique[0]} (average)"
+        else:
+            metric_name = "mixed (average)"
+
+        return (metric_name, avg, cls.is_higher_better(metric_name))

rdagent/app/finetune/llm/ui/components.py

@@ -0,0 +1,721 @@
+"""
+FT UI Components - Hierarchical Event Renderers
+"""
+
+import re
+from pathlib import Path
+from typing import Any
+
+import plotly.graph_objects as go
+import streamlit as st
+
+from rdagent.app.finetune.llm.ui.benchmarks import get_core_metric_score
+from rdagent.app.finetune.llm.ui.config import ICONS
+from rdagent.app.finetune.llm.ui.data_loader import Event, EvoLoop, Loop, Session
+
+
+def convert_latex_for_streamlit(text: str) -> str:
+    """Convert LaTeX syntax to Streamlit-compatible format.
+
+    Streamlit uses $...$ and $$...$$ for LaTeX rendering.
+    This converts \(...\) and \[...\] to the Streamlit format.
+    """
+    if not text:
+        return text
+    # Convert \(...\) to $...$
+    text = text.replace(r"\(", "$").replace(r"\)", "$")
+    # Convert \[...\] to $$...$$
+    text = text.replace(r"\[", "$$").replace(r"\]", "$$")
+    return text
+
+
+def format_duration(seconds: float | None) -> str:
+    if seconds is None:
+        return ""
+    if seconds < 60:
+        return f"{seconds:.1f}s"
+    minutes = int(seconds // 60)
+    secs = seconds % 60
+    return f"{minutes}m {secs:.0f}s"
+
+
+def render_session(session: Session, show_types: list[str]) -> None:
+    """Render full session with hierarchy"""
+    # Init events (before any loop)
+    if session.init_events:
+        filtered = [e for e in session.init_events if e.type in show_types]
+        if filtered:
+            with st.expander("🚀 **Initialization**", expanded=False):
+                for event in filtered:
+                    render_event(event)
+
+    # Loops
+    for loop_id in sorted(session.loops.keys()):
+        loop = session.loops[loop_id]
+        render_loop(loop, show_types)
+
+
+def render_loop(loop: Loop, show_types: list[str]) -> None:
+    """Render a single loop with lazy loading"""
+    # 1. Coding stage results
+    evo_results = []
+    for evo in loop.coding.values():
+        if evo.success is True:
+            evo_results.append("✓")
+        elif evo.success is False:
+            evo_results.append("✗")
+    coding_str = f"💻{''.join(evo_results)}" if evo_results else ""
+
+    # 2. Running stage results
+    runner_success = None
+    benchmark_score = None
+    for event in loop.runner:
+        # Docker (Full Train) result - check exit_code, not LLM evaluation
+        if event.type == "docker_exec" and "Full Train" in event.title and event.success is not None:
+            runner_success = event.success
+        # Benchmark score - use core metric from processor
+        if event.type == "feedback" and "Benchmark Result" in event.title:
+            content = event.content
+            if isinstance(content, dict):
+                benchmark_name = content.get("benchmark_name", "")
+                accuracy_summary = content.get("accuracy_summary", {})
+                if isinstance(accuracy_summary, dict) and accuracy_summary:
+                    result = get_core_metric_score(benchmark_name, accuracy_summary)
+                    if result is not None:
+                        _, benchmark_score, _ = result
+
+    # 3. Get feedback decision for benchmark score coloring
+    feedback_decision = None
+    for event in loop.feedback:
+        if event.type == "feedback" and "Feedback:" in event.title:
+            feedback_decision = event.success
+            break
+
+    # 4. Build title string (only show existing stages)
+    parts = []
+    if coding_str:
+        parts.append(coding_str)
+    if runner_success is not None:
+        runner_str = "🏃✓" if runner_success else "🏃✗"
+        parts.append(runner_str)
+    # Show benchmark score with emoji based on feedback decision
+    if benchmark_score is not None:
+        if feedback_decision is True:
+            parts.append(f"✅📊{benchmark_score:.2f}")
+        elif feedback_decision is False:
+            parts.append(f"❌📊{benchmark_score:.2f}")
+        else:
+            parts.append(f"📊{benchmark_score:.2f}")
+
+    result_str = " ".join(parts) if parts else ""
+
+    loop_key = f"loop_{loop.loop_id}_loaded"
+    with st.expander(f"🔄 **Loop {loop.loop_id}** {result_str}", expanded=False):
+        if not st.session_state.get(loop_key, False):
+            # Lazy load: show button first
+            if st.button("📥 Load Content", key=f"load_{loop.loop_id}"):
+                st.session_state[loop_key] = True
+                st.rerun()
+        else:
+            # Render actual content
+            _render_loop_content(loop, show_types)
+
+
+def _render_loop_content(loop: Loop, show_types: list[str]) -> None:
+    """Render loop content (called after lazy load)"""
+    # Exp Gen
+    if loop.exp_gen:
+        filtered = [e for e in loop.exp_gen if e.type in show_types]
+        if filtered:
+            st.markdown("#### 🧪 Experiment Generation")
+            for event in filtered:
+                render_event(event)
+
+    # Coding (Evo Loops)
+    if loop.coding:
+        st.markdown("#### 💻 Coding")
+        for evo_id in sorted(loop.coding.keys()):
+            evo = loop.coding[evo_id]
+            render_evo_loop(evo, show_types)
+
+    # Runner
+    if loop.runner:
+        filtered = [e for e in loop.runner if e.type in show_types]
+        if filtered:
+            st.markdown("#### 🏃 Running(Full Train)")
+            for event in filtered:
+                render_event(event)
+
+    # Feedback
+    if loop.feedback:
+        filtered = [e for e in loop.feedback if e.type in show_types]
+        if filtered:
+            st.markdown("#### 📊 Feedback")
+            for event in filtered:
+                render_event(event)
+
+
+def render_evo_loop(evo: EvoLoop, show_types: list[str]) -> None:
+    """Render evolution loop"""
+    filtered = [e for e in evo.events if e.type in show_types]
+    if not filtered:
+        return
+
+    status = "🟢" if evo.success else "🔴" if evo.success is False else "⚪"
+    with st.expander(f"{status} Evo {evo.evo_id}", expanded=False):
+        for event in filtered:
+            render_event(event)
+
+
+def render_event(event: Event) -> None:
+    """Render a single event"""
+    icon = ICONS.get(event.type, "📌")
+    duration_str = f" ({format_duration(event.duration)})" if event.duration else ""
+
+    status = ""
+    if event.success is True:
+        status = "🟢 "
+    elif event.success is False:
+        status = "🔴 "
+
+    title = f"{event.time_str} {icon} {status}{event.title}{duration_str}"
+
+    renderers = {
+        "scenario": render_scenario,
+        "llm_call": render_llm_call,
+        "template": render_template,
+        "experiment": render_experiment,
+        "code": render_code,
+        "docker_exec": render_docker_exec,
+        "evaluator": render_docker_exec,  # Reuse docker_exec renderer for evaluator feedback
+        "feedback": render_feedback,
+        "token": render_token,
+        "time": render_time_info,
+        "settings": render_settings,
+        "hypothesis": render_hypothesis,
+        "dataset_selection": render_dataset_selection,
+    }
+
+    renderer = renderers.get(event.type, render_generic)
+    with st.expander(title, expanded=False):
+        # Pass event.title to docker_exec/evaluator renderers for context-aware labels
+        if event.type in ("docker_exec", "evaluator"):
+            renderer(event.content, event.title)
+        else:
+            renderer(event.content)
+
+
+def render_scenario(content: Any) -> None:
+    """Render scenario details (main info shown in page header, this shows extras)."""
+    import json
+
+    # 1. User target scenario
+    if hasattr(content, "user_target_scenario") and content.user_target_scenario:
+        st.markdown(f"**Target Scenario:** {content.user_target_scenario}")
+
+    # 2. Benchmark description
+    if hasattr(content, "benchmark_description") and content.benchmark_description:
+        st.markdown(f"**Benchmark Description:** {content.benchmark_description}")
+
+    # 3. Full timeout
+    if hasattr(content, "real_full_timeout"):
+        try:
+            timeout_hours = content.real_full_timeout() / 60 / 60
+            st.markdown(f"**Full Train Timeout:** {timeout_hours:.2f} hours")
+        except Exception:
+            pass
+
+    # 4. Device info - formatted nicely
+    if hasattr(content, "device_info") and content.device_info:
+        device = content.device_info
+        # Parse string to dict if needed
+        if isinstance(device, str):
+            try:
+                device = json.loads(device)
+            except json.JSONDecodeError:
+                st.markdown(f"**Device:** `{device}`")
+                device = None
+        if isinstance(device, dict):
+            parts = []
+            # Runtime info
+            runtime = device.get("runtime", {})
+            if runtime.get("python_version"):
+                parts.append(f"🐍 Python `{runtime['python_version'].split()[0]}`")
+            if runtime.get("os"):
+                parts.append(f"💻 {runtime['os']}")
+            # GPU info
+            gpu_info = device.get("gpu", {})
+            gpus = gpu_info.get("gpus", [])
+            if gpus:
+                gpu_name = gpus[0].get("name", "Unknown")
+                gpu_mem_gb = gpus[0].get("memory_total_gb", 0)
+                if len(gpus) > 1:
+                    parts.append(f"🎮 {len(gpus)}x {gpu_name} ({gpu_mem_gb}GB)")
+                else:
+                    parts.append(f"🎮 {gpu_name} ({gpu_mem_gb}GB)")
+            if parts:
+                st.markdown(" · ".join(parts))
+
+    # 5. Model info (detailed specs)
+    if hasattr(content, "model_info") and content.model_info:
+        model_info = content.model_info
+        if isinstance(model_info, dict) and model_info:
+            with st.expander("Model Info", expanded=False):
+                # Show key specs in a readable format
+                if "specs" in model_info and model_info["specs"]:
+                    st.markdown("**Specs:**")
+                    st.code(model_info["specs"], language="text", wrap_lines=True)
+                # Show other fields
+                other_info = {k: v for k, v in model_info.items() if k != "specs" and v}
+                if other_info:
+                    st.json(other_info)
+
+    # 6. Memory report (estimation based on hardware and model)
+    if hasattr(content, "memory_report") and content.memory_report:
+        with st.expander("Memory Estimation", expanded=False):
+            st.code(content.memory_report, language="text", wrap_lines=True)
+
+
+def render_dataset_selection(content: Any) -> None:
+    if not isinstance(content, dict):
+        st.json(content) if content else st.info("No content")
+        return
+
+    selected = content.get("selected_datasets", [])
+    total = content.get("total_datasets", 0)
+    reasoning = content.get("reasoning", "")
+
+    if selected:
+        st.markdown(f"**Selected ({len(selected)}/{total}):** " + ", ".join(f"`{ds}`" for ds in selected))
+
+    if reasoning:
+        with st.expander("Selection Reasoning", expanded=True):
+            st.markdown(reasoning)
+
+
+def render_hypothesis(content: Any) -> None:
+    """Render hypothesis content (Base Model shown in page header, not here)."""
+    if hasattr(content, "hypothesis") and content.hypothesis:
+        st.markdown("**Hypothesis:**")
+        st.markdown(content.hypothesis)
+    if hasattr(content, "reason") and content.reason:
+        with st.expander("Reason", expanded=False):
+            st.markdown(content.reason)
+
+
+def render_settings(content: Any) -> None:
+    if isinstance(content, dict):
+        st.json(content)
+    else:
+        st.code(str(content), wrap_lines=True)
+
+
+def render_llm_call(content: Any) -> None:
+    if not isinstance(content, dict):
+        st.json(content) if content else st.info("No content")
+        return
+
+    if content.get("start") and content.get("end"):
+        duration = (content["end"] - content["start"]).total_seconds()
+        st.caption(f"Duration: {format_duration(duration)}")
+
+    # Check if markdown rendering is enabled
+    render_md = st.session_state.get("render_markdown_toggle", False)
+
+    system = content.get("system", "")
+    if system:
+        with st.expander("System Prompt", expanded=False):
+            if render_md:
+                st.markdown(system)
+            else:
+                st.code(system, language="text", line_numbers=True, wrap_lines=True)
+
+    user = content.get("user", "")
+    if user:
+        with st.expander("User Prompt", expanded=False):
+            if render_md:
+                st.markdown(user)
+            else:
+                st.code(user, language="text", line_numbers=True, wrap_lines=True)
+
+    resp = content.get("resp", "")
+    if resp:
+        st.markdown("**Response:**")
+        if render_md:
+            st.markdown(resp)
+        elif resp.strip().startswith("{") or resp.strip().startswith("["):
+            st.code(resp, language="json", line_numbers=True, wrap_lines=True)
+        elif resp.strip().startswith("```"):
+            st.markdown(resp)
+        else:
+            st.code(resp, language="text", line_numbers=True, wrap_lines=True)
+
+
+def render_template(content: Any) -> None:
+    if not isinstance(content, dict):
+        st.json(content) if content else st.info("No content")
+        return
+
+    uri = content.get("uri", "")
+    st.caption(f"URI: `{uri}`")
+
+    context = content.get("context", {})
+    if context:
+        with st.expander("Context Variables", expanded=False):
+            st.json(context)
+
+    template = content.get("template", "")
+    if template:
+        with st.expander("Template", expanded=False):
+            st.code(template, language="text", line_numbers=True, wrap_lines=True)
+
+    rendered = content.get("rendered", "")
+    if rendered:
+        with st.expander("Rendered", expanded=True):
+            st.code(rendered, language="text", line_numbers=True, wrap_lines=True)
+
+
+def render_experiment(content: Any) -> None:
+    """Render experiment tasks (Base Model and Datasets shown in page header, not here)."""
+    if isinstance(content, list):
+        for i, task in enumerate(content):
+            if len(content) > 1:
+                st.markdown(f"**Task {i}**")
+
+            if hasattr(task, "description") and task.description:
+                st.markdown("**Description:**")
+                st.markdown(task.description)
+    else:
+        st.json(content) if content else st.info("No content")
+
+
+def render_code(content: Any) -> None:
+    if not isinstance(content, list):
+        st.info("No code available")
+        return
+
+    for i, ws in enumerate(content):
+        if not hasattr(ws, "file_dict") or not ws.file_dict:
+            continue
+
+        if len(content) > 1:
+            st.markdown(f"**Workspace {i}**")
+
+        for filename, code in ws.file_dict.items():
+            lang = "yaml" if filename.endswith((".yaml", ".yml")) else "python"
+            with st.expander(filename, expanded=False):
+                st.code(code, language=lang, line_numbers=True, wrap_lines=True)
+
+
+def _extract_evaluator_name(title: str) -> str:
+    """Extract evaluator name from event title like 'Eval (Data Processing) ✓'."""
+    match = re.search(r"\(([^)]+)\)", title)
+    return match.group(1) if match else ""
+
+
+def _render_single_feedback(fb: Any, evaluator_name: str = "") -> None:
+    """Render a single CoSTEERSingleFeedback object.
+
+    Structure:
+    - execution: LLM-generated execution summary (what happened, success/failure reason)
+    - raw_execution: Raw script stdout/stderr output
+    - return_checking: LLM-generated data quality assessment
+    - code: LLM-generated code improvement suggestions
+    """
+    decision = getattr(fb, "final_decision", None)
+    if decision is True:
+        st.success("Execution: PASS")
+    elif decision is False:
+        st.error("Execution: FAIL")
+
+    # 1. Execution Summary (LLM-generated)
+    execution = getattr(fb, "execution", "")
+    if execution:
+        label = f"{evaluator_name} Summary" if evaluator_name else "Execution Summary"
+        with st.expander(label, expanded=True):
+            st.code(execution, language="text", line_numbers=True, wrap_lines=True)
+
+    # 2. Raw Execution Log (script stdout)
+    raw_execution = getattr(fb, "raw_execution", "")
+    if raw_execution:
+        with st.expander("Raw Output (stdout)", expanded=False):
+            st.code(raw_execution, language="text", line_numbers=True, wrap_lines=True)
+
+    # 3. Data Quality Check (LLM-generated)
+    return_checking = getattr(fb, "return_checking", "")
+    if return_checking:
+        with st.expander("Data Quality Check", expanded=False):
+            st.code(return_checking, language="text", line_numbers=True, wrap_lines=True)
+
+    # 4. Code Improvement Suggestions (LLM-generated, often very long)
+    code_fb = getattr(fb, "code", "")
+    if code_fb:
+        with st.expander("Code Improvement Suggestions", expanded=False):
+            # Use markdown rendering if content contains markdown formatting
+            if "**" in code_fb or "```" in code_fb or "- " in code_fb:
+                st.markdown(code_fb)
+            else:
+                st.code(code_fb, language="text", line_numbers=True, wrap_lines=True)
+
+
+def render_docker_exec(content: Any, event_title: str = "") -> None:
+    # Extract evaluator name from event title for context-aware labels
+    evaluator_name = _extract_evaluator_name(event_title)
+
+    # Docker run raw output (dict with exit_code/stdout)
+    if isinstance(content, dict) and ("exit_code" in content or "stdout" in content or "success" in content):
+        # Show workspace ID if available (only the UUID part)
+        workspace_path = content.get("workspace_path")
+        if workspace_path:
+            workspace_id = Path(workspace_path).name
+            st.caption(f"📁 `{workspace_id}`")
+
+        exit_code = content.get("exit_code")
+        success = content.get("success")
+        if exit_code is not None:
+            if exit_code == 0:
+                st.success(f"Exit code: {exit_code}")
+            else:
+                st.error(f"Exit code: {exit_code}")
+        elif success is not None:
+            if success:
+                st.success("Execution: PASS")
+            else:
+                st.error("Execution: FAIL")
+
+        stdout = content.get("stdout", "")
+        if stdout:
+            label = f"{evaluator_name} Output" if evaluator_name else "Execution Output"
+            with st.expander(label, expanded=True):
+                st.code(stdout, language="text", line_numbers=True, wrap_lines=True)
+        return
+
+    # CoSTEERMultiFeedback (has feedback_list)
+    if hasattr(content, "feedback_list"):
+        for i, fb in enumerate(content.feedback_list):
+            if len(content.feedback_list) > 1:
+                st.markdown(f"**Feedback {i}**")
+            _render_single_feedback(fb, evaluator_name)
+        return
+
+    # Single CoSTEERSingleFeedback (has final_decision)
+    if hasattr(content, "final_decision"):
+        _render_single_feedback(content, evaluator_name)
+        return
+
+    # FTExperiment (runner result)
+    if hasattr(content, "sub_workspace_list"):
+        for ws in content.sub_workspace_list:
+            if not hasattr(ws, "running_info") or ws.running_info is None:
+                continue
+
+            info = ws.running_info
+            running_time = getattr(info, "running_time", None)
+            if running_time:
+                st.metric("Running Time", f"{running_time:.1f}s")
+
+            stdout = getattr(info, "stdout", "")
+            if stdout:
+                with st.expander("Full Train Log", expanded=True):
+                    st.code(stdout, language="text", line_numbers=True, wrap_lines=True)
+
+            result = getattr(info, "result", {})
+            if result:
+                render_training_result(result)
+        return
+
+    st.json(content) if content else st.info("No content")
+
+
+def render_feedback(content: Any) -> None:
+    # Handle benchmark result (dict with accuracy_summary)
+    if isinstance(content, dict) and "accuracy_summary" in content:
+        render_benchmark_result(content)
+        return
+
+    col1, col2, col3 = st.columns(3)
+    with col1:
+        decision = getattr(content, "decision", None)
+        if decision is not None:
+            st.metric("Decision", "Accept" if decision else "Reject")
+    with col2:
+        acceptable = getattr(content, "acceptable", None)
+        if acceptable is not None:
+            st.metric("Acceptable", "Yes" if acceptable else "No")
+    with col3:
+        error_type = getattr(content, "observations", None)
+        if error_type:
+            st.metric("Error Type", error_type)
+
+    # FT scenario only uses code_change_summary (observations, hypothesis_evaluation,
+    # new_hypothesis, eda_improvement are DS scenario specific)
+    fields = [
+        ("code_change_summary", "Code Change Summary"),
+    ]
+
+    for attr, label in fields:
+        value = getattr(content, attr, None)
+        if value:
+            with st.expander(label, expanded=False):
+                st.markdown(value)
+
+    reason = getattr(content, "reason", None)
+    if reason:
+        with st.expander("Reason (Full Details)", expanded=True):
+            st.code(reason, language="text", line_numbers=True, wrap_lines=True)
+
+    exception = getattr(content, "exception", None)
+    if exception:
+        st.error(f"Exception: {exception}")
+
+
+def render_token(content: Any) -> None:
+    if isinstance(content, dict):
+        col1, col2, col3 = st.columns(3)
+        with col1:
+            st.metric("Prompt", content.get("prompt_tokens", 0))
+        with col2:
+            st.metric("Completion", content.get("completion_tokens", 0))
+        with col3:
+            st.metric("Total", content.get("total_tokens", 0))
+    else:
+        st.json(content) if content else st.info("No content")
+
+
+def render_time_info(content: Any) -> None:
+    if isinstance(content, dict):
+        for k, v in content.items():
+            st.metric(k, f"{v:.1f}s" if isinstance(v, (int, float)) else str(v))
+    else:
+        st.json(content) if content else st.info("No content")
+
+
+def render_generic(content: Any) -> None:
+    if hasattr(content, "__dict__"):
+        st.json(vars(content))
+    elif content:
+        st.json(content)
+    else:
+        st.info("No content")
+
+
+def render_training_result(result: dict) -> None:
+    training_metrics = result.get("training_metrics", {})
+    loss_history = training_metrics.get("loss_history", {})
+
+    # loss_history is Dict[str, List[Dict]] with "train" and "eval" keys
+    train_history = loss_history.get("train", []) if isinstance(loss_history, dict) else []
+    if train_history:
+        fig = go.Figure()
+        steps = [entry.get("step", i) for i, entry in enumerate(train_history)]
+        losses = [entry.get("loss", 0) for entry in train_history]
+        fig.add_trace(go.Scatter(x=steps, y=losses, mode="lines+markers", name="Loss"))
+        fig.update_layout(title="Training Loss", xaxis_title="Step", yaxis_title="Loss", height=300)
+        st.plotly_chart(fig, use_container_width=True)
+
+        col1, col2 = st.columns(2)
+        initial_loss = training_metrics.get("initial_loss")
+        final_loss = training_metrics.get("final_loss")
+        if initial_loss:
+            col1.metric("Initial Loss", f"{initial_loss:.4f}")
+        if final_loss:
+            col2.metric("Final Loss", f"{final_loss:.4f}")
+
+    # Validation benchmark ([:100]) - used for SOTA judgment
+    benchmark = result.get("benchmark", {})
+    if benchmark:
+        st.markdown("**Validation Benchmark**")
+        # Detect format: old format has "accuracy_summary" at top level,
+        # new format has benchmark names as keys with nested accuracy_summary
+        if "accuracy_summary" in benchmark:
+            # Old format: {accuracy_summary: {...}, error_samples: [...]}
+            accuracy_summary = benchmark.get("accuracy_summary", {})
+            if accuracy_summary:
+                rows = [{"dataset": ds, **metrics} for ds, metrics in accuracy_summary.items()]
+                st.dataframe(rows)
+        else:
+            # New format: {bm_name: {accuracy_summary: {...}}, ...}
+            for bm_name, bm_result in benchmark.items():
+                if isinstance(bm_result, dict) and "accuracy_summary" in bm_result:
+                    st.markdown(f"*{bm_name}:*")
+                    accuracy_summary = bm_result.get("accuracy_summary", {})
+                    if accuracy_summary:
+                        rows = [{"dataset": ds, **metrics} for ds, metrics in accuracy_summary.items()]
+                        st.dataframe(rows)
+
+    # Test benchmark ([100:200]) - frontend display only, not visible to agent
+    benchmark_test = result.get("benchmark_test", {})
+    if benchmark_test and benchmark_test != benchmark:  # Avoid duplicate display for small datasets
+        st.markdown("**Test Benchmark**")
+        if "accuracy_summary" in benchmark_test:
+            accuracy_summary = benchmark_test.get("accuracy_summary", {})
+            if accuracy_summary:
+                rows = [{"dataset": ds, **metrics} for ds, metrics in accuracy_summary.items()]
+                st.dataframe(rows)
+        else:
+            for bm_name, bm_result in benchmark_test.items():
+                if isinstance(bm_result, dict) and "accuracy_summary" in bm_result:
+                    st.markdown(f"*{bm_name}:*")
+                    accuracy_summary = bm_result.get("accuracy_summary", {})
+                    if accuracy_summary:
+                        rows = [{"dataset": ds, **metrics} for ds, metrics in accuracy_summary.items()]
+                        st.dataframe(rows)
+
+
+def render_benchmark_result(content: dict) -> None:
+    """Render benchmark evaluation result"""
+    import pandas as pd
+
+    benchmark_name = content.get("benchmark_name", "Unknown")
+    st.markdown(f"**Benchmark: {benchmark_name}**")
+
+    # Accuracy summary table
+    # accuracy_summary is a dict: {dataset_name: {metric: value, ...}, ...}
+    accuracy_summary = content.get("accuracy_summary", {})
+    if accuracy_summary and isinstance(accuracy_summary, dict):
+        st.markdown("**Accuracy Summary:**")
+        # Convert dict {dataset: {metric: value}} to list of dicts for dataframe
+        rows = []
+        for ds, metrics in accuracy_summary.items():
+            row = {"dataset": ds, **metrics}
+            rows.append(row)
+
+        # Create DataFrame and reorder columns
+        df = pd.DataFrame(rows)
+        cols = ["dataset"] + [c for c in df.columns if c != "dataset"]
+        df = df[cols]
+        st.dataframe(df)
+
+    # Error samples
+    error_samples = content.get("error_samples", [])
+    if error_samples:
+        with st.expander(f"Error Samples ({len(error_samples)})", expanded=False):
+            for i, sample in enumerate(error_samples):
+                with st.expander(f"Sample {i+1} (Gold: {sample.get('gold', 'N/A')})", expanded=False):
+                    st.markdown(
+                        '<div style="font-size: 0.85em;">',
+                        unsafe_allow_html=True,
+                    )
+                    st.markdown("**Question:**")
+                    st.markdown(convert_latex_for_streamlit(sample.get("question", "N/A")))
+                    st.markdown("---")
+                    st.markdown(f"**Gold:** `{sample.get('gold', 'N/A')}`")
+                    st.markdown("---")
+                    st.markdown("**Model Output:**")
+                    st.markdown(convert_latex_for_streamlit(sample.get("model_output", "N/A")))
+                    st.markdown("</div>", unsafe_allow_html=True)
+
+
+def render_summary(summary: dict) -> None:
+    col1, col2, col3, col4 = st.columns(4)
+    with col1:
+        st.metric("Loops", summary.get("loop_count", 0))
+    with col2:
+        st.metric("LLM Calls", summary.get("llm_call_count", 0))
+    with col3:
+        llm_time = summary.get("llm_total_time", 0)
+        st.metric("LLM Time", format_duration(llm_time))
+    with col4:
+        success = summary.get("docker_success", 0)
+        fail = summary.get("docker_fail", 0)
+        st.metric("Executions", f"{success}✓ / {fail}✗")

rdagent/app/finetune/llm/ui/config.py

@@ -0,0 +1,69 @@
+"""
+FT UI Configuration Constants
+
+Centralized configuration for FT Timeline Viewer.
+"""
+
+from typing import Literal
+
+# Event type definition
+EventType = Literal[
+    "scenario",
+    "llm_call",
+    "template",
+    "experiment",
+    "code",
+    "docker_exec",
+    "evaluator",  # Evaluator feedback (separate from docker_exec)
+    "feedback",
+    "token",
+    "time",
+    "settings",
+    "hypothesis",
+    "dataset_selection",
+]
+
+# Event type icons
+ICONS = {
+    "scenario": "🎯",
+    "llm_call": "💬",
+    "template": "📋",
+    "experiment": "🧪",
+    "code": "📄",
+    "docker_exec": "🐳",
+    "evaluator": "📝",  # Evaluator feedback icon
+    "feedback": "📊",
+    "token": "🔢",
+    "time": "⏱️",
+    "settings": "⚙️",
+    "hypothesis": "💡",
+    "dataset_selection": "📂",
+}
+
+# Evaluator configuration mapping (name, default_stage)
+EVALUATOR_CONFIG = {
+    "FTDataEvaluator": ("Data Processing", "coding"),
+    "FTCoderEvaluator": ("Micro-batch Test", "coding"),
+    "FTRunnerEvaluator": ("Full Train", "runner"),
+}
+
+# Always visible event types
+ALWAYS_VISIBLE_TYPES = [
+    "scenario",
+    "dataset_selection",
+    "hypothesis",
+    "llm_call",
+    "experiment",
+    "code",
+    "docker_exec",
+    "evaluator",
+    "feedback",
+]
+
+# Optional event types with toggle config (label, default_enabled)
+OPTIONAL_TYPES = {
+    "template": ("📋 Template", False),
+    "token": ("🔢 Token", False),
+    "time": ("⏱️ Time", False),
+    "settings": ("⚙️ Settings", False),
+}

rdagent/app/finetune/llm/ui/data_loader.py

@@ -0,0 +1,455 @@
+"""
+FT UI Data Loader
+Load pkl logs and convert to hierarchical timeline structure
+"""
+
+import re
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+import streamlit as st
+
+from rdagent.app.finetune.llm.ui.config import EVALUATOR_CONFIG, EventType
+from rdagent.log.storage import FileStorage
+
+
+@dataclass
+class Event:
+    """Timeline event"""
+
+    type: EventType
+    timestamp: datetime
+    tag: str
+    title: str
+    content: Any
+    loop_id: int | None = None
+    evo_id: int | None = None
+    stage: str = ""
+    duration: float | None = None
+    success: bool | None = None
+
+    @property
+    def time_str(self) -> str:
+        return self.timestamp.strftime("%H:%M:%S")
+
+
+@dataclass
+class EvoLoop:
+    """Evolution loop containing events"""
+
+    evo_id: int
+    events: list[Event] = field(default_factory=list)
+    success: bool | None = None
+
+
+@dataclass
+class Loop:
+    """Main loop containing stages"""
+
+    loop_id: int
+    exp_gen: list[Event] = field(default_factory=list)
+    coding: dict[int, EvoLoop] = field(default_factory=dict)  # evo_id -> EvoLoop
+    runner: list[Event] = field(default_factory=list)
+    feedback: list[Event] = field(default_factory=list)
+
+
+@dataclass
+class Session:
+    """Session containing init events and loops"""
+
+    init_events: list[Event] = field(default_factory=list)
+    loops: dict[int, Loop] = field(default_factory=dict)  # loop_id -> Loop
+
+
+def extract_loop_id(tag: str) -> int | None:
+    match = re.search(r"Loop_(\d+)", tag)
+    return int(match.group(1)) if match else None
+
+
+def extract_evo_id(tag: str) -> int | None:
+    match = re.search(r"evo_loop_(\d+)", tag)
+    return int(match.group(1)) if match else None
+
+
+def extract_stage(tag: str) -> str:
+    if "direct_exp_gen" in tag:
+        return "exp_gen"
+    if "coding" in tag:
+        return "coding"
+    if "running" in tag:  # Note: tag uses "running", not "runner"
+        return "runner"
+    if "feedback" in tag:
+        return "feedback"
+    return ""
+
+
+def get_valid_sessions(log_folder: Path) -> list[str]:
+    if not log_folder.exists():
+        return []
+    sessions = []
+    for d in log_folder.iterdir():
+        if d.is_dir() and d.joinpath("__session__").exists():
+            sessions.append(d.name)
+    return sorted(sessions, reverse=True)
+
+
+def parse_event(tag: str, content: Any, timestamp: datetime) -> Event | None:
+    loop_id = extract_loop_id(tag)
+    evo_id = extract_evo_id(tag)
+    stage = extract_stage(tag)
+
+    # Scenario
+    if tag == "scenario":
+        model = getattr(content, "base_model", "Unknown")
+        return Event(type="scenario", timestamp=timestamp, tag=tag, title=f"Scenario: {model}", content=content)
+
+    # Dataset selection
+    if "dataset_selection" in tag:
+        selected = content.get("selected_datasets", []) if isinstance(content, dict) else []
+        total = content.get("total_datasets", 0) if isinstance(content, dict) else 0
+        return Event(
+            type="dataset_selection",
+            timestamp=timestamp,
+            tag=tag,
+            title=f"Dataset Selection: {len(selected)}/{total}",
+            content=content,
+        )
+
+    # Settings
+    if "SETTINGS" in tag:
+        name = tag.replace("_SETTINGS", "").replace("SETTINGS", "")
+        return Event(type="settings", timestamp=timestamp, tag=tag, title=f"Settings: {name}", content=content)
+
+    # Hypothesis
+    if tag == "hypothesis" or (loop_id is not None and "hypothesis" in tag):
+        return Event(
+            type="hypothesis",
+            timestamp=timestamp,
+            tag=tag,
+            title="Hypothesis",
+            content=content,
+            loop_id=loop_id,
+            stage="exp_gen",
+        )
+
+    # LLM Call
+    if "debug_llm" in tag:
+        if isinstance(content, dict) and ("user" in content or "system" in content):
+            duration = None
+            if content.get("start") and content.get("end"):
+                duration = (content["end"] - content["start"]).total_seconds()
+            return Event(
+                type="llm_call",
+                timestamp=timestamp,
+                tag=tag,
+                title="LLM Call",
+                content=content,
+                loop_id=loop_id,
+                evo_id=evo_id,
+                stage=stage,
+                duration=duration,
+            )
+
+    # Template
+    if "debug_tpl" in tag:
+        if isinstance(content, dict) and "uri" in content:
+            uri = content.get("uri", "")
+            tpl_name = uri.split(":")[-1] if ":" in uri else uri
+            return Event(
+                type="template",
+                timestamp=timestamp,
+                tag=tag,
+                title=f"Template: {tpl_name}",
+                content=content,
+                loop_id=loop_id,
+                evo_id=evo_id,
+                stage=stage,
+            )
+
+    # Experiment generation
+    if "experiment generation" in tag:
+        task_count = len(content) if isinstance(content, list) else 1
+        return Event(
+            type="experiment",
+            timestamp=timestamp,
+            tag=tag,
+            title=f"Experiment ({task_count} task)",
+            content=content,
+            loop_id=loop_id,
+            stage=stage,
+        )
+
+    # Evolving code
+    if "evolving code" in tag:
+        file_count = 0
+        if isinstance(content, list):
+            for ws in content:
+                if hasattr(ws, "file_dict"):
+                    file_count += len(ws.file_dict)
+        return Event(
+            type="code",
+            timestamp=timestamp,
+            tag=tag,
+            title=f"Code ({file_count} files)",
+            content=content,
+            loop_id=loop_id,
+            evo_id=evo_id,
+            stage=stage or "coding",
+        )
+
+    # Benchmark execution (Docker or Conda) - must check before generic docker_run/conda_run
+    if "docker_run.Benchmark" in tag or "conda_run.Benchmark" in tag:
+        benchmark_name = content.get("benchmark_name", "Unknown") if isinstance(content, dict) else "Unknown"
+        exit_code = content.get("exit_code") if isinstance(content, dict) else None
+        success = exit_code == 0 if exit_code is not None else None
+        env_type = "Docker" if "docker_run" in tag else "Conda"
+        return Event(
+            type="docker_exec",
+            timestamp=timestamp,
+            tag=tag,
+            title=f"Benchmark ({benchmark_name}) [{env_type}] {'✓' if success else '✗' if success is False else ''}",
+            content=content,
+            loop_id=loop_id,
+            stage="runner",
+            success=success,
+        )
+
+    # Environment run (Docker or Conda, raw execution logged before LLM evaluation)
+    if "docker_run." in tag or "conda_run." in tag:
+        is_docker = "docker_run." in tag
+        tag_prefix = "docker_run." if is_docker else "conda_run."
+        class_name = tag.split(tag_prefix)[-1].split(".")[0]
+
+        # FTWorkspace unified logging - determine type from entry command
+        if class_name == "FTWorkspace":
+            entry = content.get("entry", "") if isinstance(content, dict) else ""
+            if "llamafactory-cli train" in entry:
+                # Distinguish by yaml file name: debug_train.yaml for micro-batch, train.yaml for full training
+                if "debug_train.yaml" in entry:
+                    evaluator_name, default_stage = "Micro-batch Test", "coding"
+                else:
+                    evaluator_name, default_stage = "Full Train", "runner"
+            elif "process_data" in entry.lower():
+                evaluator_name, default_stage = "Data Processing", "coding"
+            elif entry.startswith("rm "):
+                evaluator_name, default_stage = "Cleanup", "runner"
+            else:
+                evaluator_name, default_stage = "Env Run", "coding"
+        else:
+            evaluator_name, default_stage = EVALUATOR_CONFIG.get(class_name, (class_name, "coding"))
+
+        exit_code = content.get("exit_code") if isinstance(content, dict) else None
+        success = exit_code == 0 if exit_code is not None else content.get("success")
+        env_label = "Docker" if is_docker else "Conda"
+        title = f"{env_label} ({evaluator_name}) {'✓' if success else '✗' if success is False else ''}"
+        return Event(
+            type="docker_exec",
+            timestamp=timestamp,
+            tag=tag,
+            title=title,
+            content=content,
+            loop_id=loop_id,
+            evo_id=evo_id,
+            stage=stage or default_stage,
+            success=success,
+        )
+
+    # Docker execution (individual evaluator feedback, logged after LLM evaluation)
+    if "docker_exec." in tag:
+        class_name = tag.split("docker_exec.")[-1].split(".")[0]
+        evaluator_name, default_stage = EVALUATOR_CONFIG.get(class_name, (class_name, "coding"))
+        success = getattr(content, "final_decision", None)
+        title = f"Eval ({evaluator_name}) {'✓' if success else '✗' if success is False else '?'}"
+        return Event(
+            type="docker_exec",
+            timestamp=timestamp,
+            tag=tag,
+            title=title,
+            content=content,
+            loop_id=loop_id,
+            evo_id=evo_id,
+            stage=stage or default_stage,
+            success=success,
+        )
+
+    # Evaluator feedback (logged from FT evaluators with final_decision)
+    if "evaluator_feedback." in tag:
+        class_name = tag.split("evaluator_feedback.")[-1].split(".")[0]
+        evaluator_name, default_stage = EVALUATOR_CONFIG.get(class_name, (class_name, "coding"))
+        success = getattr(content, "final_decision", None)
+        title = f"Eval ({evaluator_name}) {'✓' if success else '✗' if success is False else '?'}"
+        return Event(
+            type="evaluator",  # Use dedicated evaluator type with 📝 icon
+            timestamp=timestamp,
+            tag=tag,
+            title=title,
+            content=content,
+            loop_id=loop_id,
+            evo_id=evo_id,
+            stage=stage or default_stage,
+            success=success,
+        )
+
+    # Final feedback
+    if "feedback.feedback" in tag or (tag.endswith(".feedback") and "evo_loop" not in tag):
+        decision = getattr(content, "decision", None)
+        return Event(
+            type="feedback",
+            timestamp=timestamp,
+            tag=tag,
+            title=f"Feedback: {'Accept' if decision else 'Reject'}",
+            content=content,
+            loop_id=loop_id,
+            stage="feedback",
+            success=decision,
+        )
+
+    # Benchmark result (supports benchmark_result, benchmark_result.validation, benchmark_result.test)
+    if "benchmark_result" in tag:
+        benchmark_name = content.get("benchmark_name", "Unknown") if isinstance(content, dict) else "Unknown"
+        accuracy = content.get("accuracy_summary", {}) if isinstance(content, dict) else {}
+        # Extract split from tag or content
+        split = content.get("split", "") if isinstance(content, dict) else ""
+        if not split and "." in tag:
+            split = tag.split(".")[-1]  # e.g., "validation" or "test" from "benchmark_result.validation"
+        split_label = f" [{split.title()}]" if split and split != "default" else ""
+        return Event(
+            type="feedback",
+            timestamp=timestamp,
+            tag=tag,
+            title=f"Benchmark Result{split_label} ({benchmark_name}: {len(accuracy)} datasets)",
+            content=content,
+            loop_id=loop_id,
+            stage="runner",
+        )
+
+    # Runner result
+    if "runner result" in tag:
+        return Event(
+            type="docker_exec",
+            timestamp=timestamp,
+            tag=tag,
+            title="Full Train",
+            content=content,
+            loop_id=loop_id,
+            stage="runner",
+        )
+
+    # Token cost
+    if "token_cost" in tag:
+        if isinstance(content, dict):
+            total = content.get("total_tokens", 0)
+            return Event(
+                type="token",
+                timestamp=timestamp,
+                tag=tag,
+                title=f"Token: {total}",
+                content=content,
+                loop_id=loop_id,
+                evo_id=evo_id,
+                stage=stage,
+            )
+
+    # Time info
+    if "time_info" in tag:
+        return Event(
+            type="time", timestamp=timestamp, tag=tag, title="Time Info", content=content, loop_id=loop_id, stage=stage
+        )
+
+    return None
+
+
+@st.cache_data(ttl=300, hash_funcs={Path: str})
+def load_ft_session(log_path: Path) -> Session:
+    """Load events into hierarchical session structure"""
+    session = Session()
+    storage = FileStorage(log_path)
+
+    events = []
+    for msg in storage.iter_msg():
+        if not msg.tag:
+            continue
+        event = parse_event(msg.tag, msg.content, msg.timestamp)
+        if event:
+            events.append(event)
+
+    # Sort by timestamp
+    events.sort(key=lambda e: e.timestamp)
+
+    # Organize into hierarchy
+    for event in events:
+        if event.loop_id is None:
+            session.init_events.append(event)
+            continue
+
+        # Ensure loop exists
+        if event.loop_id not in session.loops:
+            session.loops[event.loop_id] = Loop(loop_id=event.loop_id)
+        loop = session.loops[event.loop_id]
+
+        # Place event in appropriate stage
+        if event.stage == "exp_gen":
+            loop.exp_gen.append(event)
+        elif event.stage == "coding":
+            if event.evo_id is not None:
+                if event.evo_id not in loop.coding:
+                    loop.coding[event.evo_id] = EvoLoop(evo_id=event.evo_id)
+                evo = loop.coding[event.evo_id]
+                evo.events.append(event)
+                # Use evaluator feedback (final_decision) for evo success, fallback to docker_exec
+                if event.type in ("evaluator", "docker_exec") and event.success is not None:
+                    if evo.success is None:
+                        evo.success = event.success
+                    else:
+                        evo.success = evo.success and event.success  # AND logic: all evaluators must pass
+            else:
+                # Coding events without evo_id go to evo 0
+                if 0 not in loop.coding:
+                    loop.coding[0] = EvoLoop(evo_id=0)
+                loop.coding[0].events.append(event)
+        elif event.stage == "runner":
+            loop.runner.append(event)
+        elif event.stage == "feedback":
+            loop.feedback.append(event)
+        else:
+            # Unknown stage - put in exp_gen
+            loop.exp_gen.append(event)
+
+    return session
+
+
+def get_summary(session: Session) -> dict:
+    """Get summary statistics"""
+    llm_calls = []
+    docker_execs = []
+
+    # Collect from init
+    for e in session.init_events:
+        if e.type == "llm_call":
+            llm_calls.append(e)
+        elif e.type == "docker_exec":
+            docker_execs.append(e)
+
+    # Collect from loops
+    for loop in session.loops.values():
+        for e in loop.exp_gen + loop.runner + loop.feedback:
+            if e.type == "llm_call":
+                llm_calls.append(e)
+            elif e.type == "docker_exec":
+                docker_execs.append(e)
+        for evo in loop.coding.values():
+            for e in evo.events:
+                if e.type == "llm_call":
+                    llm_calls.append(e)
+                elif e.type == "docker_exec":
+                    docker_execs.append(e)
+
+    return {
+        "loop_count": len(session.loops),
+        "llm_call_count": len(llm_calls),
+        "llm_total_time": sum(e.duration or 0 for e in llm_calls),
+        "docker_success": sum(1 for e in docker_execs if e.success is True),
+        "docker_fail": sum(1 for e in docker_execs if e.success is False),
+    }

rdagent/app/finetune/llm/ui/ft_summary.py

@@ -0,0 +1,587 @@
+"""
+FT Job Summary View
+Display summary table for all tasks in a job directory
+"""
+
+import pickle
+from pathlib import Path
+
+import pandas as pd
+import streamlit as st
+from pandas.io.formats.style import Styler
+
+from rdagent.app.finetune.llm.ui.benchmarks import get_core_metric_score
+
+
+def is_valid_task(task_path: Path) -> bool:
+    """Check if directory is a valid FT task (has __session__ subdirectory)"""
+    return task_path.is_dir() and (task_path / "__session__").exists()
+
+
+def get_loop_dirs(task_path: Path) -> list[Path]:
+    """Get sorted list of Loop directories"""
+    loops = [d for d in task_path.iterdir() if d.is_dir() and d.name.startswith("Loop_")]
+    return sorted(loops, key=lambda d: int(d.name.split("_")[1]))
+
+
+def extract_benchmark_score(loop_path: Path, split: str = "") -> tuple[str, float, bool] | None:
+    """Extract benchmark score, metric name, and direction from loop directory.
+
+    Args:
+        loop_path: Path to loop directory
+        split: Filter by split type ("validation", "test", or "" for any)
+
+    Returns:
+        (metric_name, score, higher_is_better) or None
+        - metric_name includes "(average)" suffix if multiple datasets are averaged
+        - higher_is_better: True if higher values are better
+    """
+    for pkl_file in loop_path.rglob("**/benchmark_result*/**/*.pkl"):
+        try:
+            with open(pkl_file, "rb") as f:
+                content = pickle.load(f)
+            if isinstance(content, dict):
+                # Check split filter
+                content_split = content.get("split", "")
+                if split and content_split != split:
+                    continue
+
+                benchmark_name = content.get("benchmark_name", "")
+                accuracy_summary = content.get("accuracy_summary", {})
+                if isinstance(accuracy_summary, dict) and accuracy_summary:
+                    result = get_core_metric_score(benchmark_name, accuracy_summary)
+                    if result is not None:
+                        return result
+        except Exception:
+            pass
+    return None
+
+
+def extract_benchmark_scores(loop_path: Path) -> dict[str, tuple[str, float, bool] | None]:
+    """Extract both validation and test benchmark scores from loop directory.
+
+    Returns:
+        Dict with keys "validation" and "test", each containing
+        (metric_name, score, higher_is_better) or None
+    """
+    return {
+        "validation": extract_benchmark_score(loop_path, split="validation"),
+        "test": extract_benchmark_score(loop_path, split="test"),
+    }
+
+
+def extract_baseline_score(task_path: Path) -> tuple[str, float] | None:
+    """Extract baseline benchmark score from scenario object (legacy, validation only).
+
+    Returns:
+        (metric_name, score) or None
+    """
+    scenario_dir = task_path / "scenario"
+    if not scenario_dir.exists():
+        return None
+
+    for pkl_file in scenario_dir.rglob("*.pkl"):
+        try:
+            with open(pkl_file, "rb") as f:
+                scenario = pickle.load(f)
+            baseline_score = getattr(scenario, "baseline_benchmark_score", None)
+            if baseline_score and isinstance(baseline_score, dict):
+                benchmark_name = getattr(scenario, "target_benchmark", "")
+                accuracy_summary = baseline_score.get("accuracy_summary", {})
+                if isinstance(accuracy_summary, dict) and accuracy_summary:
+                    result = get_core_metric_score(benchmark_name, accuracy_summary)
+                    if result is not None:
+                        metric_name, score, _ = result
+                        return metric_name, score
+        except Exception:
+            pass
+    return None
+
+
+def extract_baseline_scores(task_path: Path) -> dict[str, tuple[str, float, bool] | None]:
+    """Extract both validation and test baseline benchmark scores from scenario.
+
+    Returns:
+        {"validation": (metric_name, score, higher_is_better) or None,
+         "test": (metric_name, score, higher_is_better) or None}
+    """
+    scenario_dir = task_path / "scenario"
+    if not scenario_dir.exists():
+        return {"validation": None, "test": None}
+
+    for pkl_file in scenario_dir.rglob("*.pkl"):
+        try:
+            with open(pkl_file, "rb") as f:
+                scenario = pickle.load(f)
+
+            benchmark_name = getattr(scenario, "target_benchmark", "")
+            result = {"validation": None, "test": None}
+
+            # Validation score
+            baseline_val = getattr(scenario, "baseline_benchmark_score", None)
+            if baseline_val and isinstance(baseline_val, dict):
+                accuracy_summary = baseline_val.get("accuracy_summary", {})
+                if isinstance(accuracy_summary, dict) and accuracy_summary:
+                    core = get_core_metric_score(benchmark_name, accuracy_summary)
+                    if core:
+                        result["validation"] = core
+
+            # Test score (new format only)
+            baseline_test = getattr(scenario, "baseline_benchmark_score_test", None)
+            if baseline_test and isinstance(baseline_test, dict):
+                accuracy_summary = baseline_test.get("accuracy_summary", {})
+                if isinstance(accuracy_summary, dict) and accuracy_summary:
+                    core = get_core_metric_score(benchmark_name, accuracy_summary)
+                    if core:
+                        result["test"] = core
+
+            return result
+        except Exception:
+            pass
+    return {"validation": None, "test": None}
+
+
+def get_loop_status(
+    task_path: Path, loop_id: int
+) -> tuple[str, float | None, float | None, str | None, bool | None, bool]:
+    """
+    Get loop status, validation score, test score, metric name with direction arrow, feedback decision, and direction.
+    Returns: (status_str, val_score_or_none, test_score_or_none, metric_display_or_none, feedback_decision, higher_is_better)
+    Status: 'C'=Coding, 'R'=Running, 'X'=Failed, score_str=Success
+    metric_display: metric name with direction arrow (e.g., "accuracy ↑")
+    feedback_decision: True=accepted, False=rejected, None=no feedback
+    higher_is_better: True if higher values are better for this metric
+    """
+    loop_path = task_path / f"Loop_{loop_id}"
+    if not loop_path.exists():
+        return "-", None, None, None, None, True
+
+    # Check for benchmark results first (highest priority - means completed)
+    scores = extract_benchmark_scores(loop_path)
+    val_result = scores.get("validation")
+    test_result = scores.get("test")
+
+    # Fallback to old format (no split) if no validation/test found
+    if val_result is None and test_result is None:
+        legacy_result = extract_benchmark_score(loop_path, split="")
+        if legacy_result is not None:
+            val_result = legacy_result  # Treat legacy as validation
+
+    # Get feedback decision (used for both score coloring and fallback status)
+    feedback_decision = None
+    feedback_files = list(loop_path.rglob("**/feedback/**/*.pkl"))
+    for f in feedback_files:
+        try:
+            with open(f, "rb") as fp:
+                content = pickle.load(fp)
+            decision = getattr(content, "decision", None)
+            if decision is not None:
+                feedback_decision = decision
+                break
+        except Exception:
+            pass
+
+    if val_result is not None:
+        metric_name, val_score, higher_is_better = val_result
+        test_score = test_result[1] if test_result else None
+        arrow = "↑" if higher_is_better else "↓"
+        metric_display = f"{metric_name} {arrow}"
+        # Format: "val/test" or just "val" if no test
+        if test_score is not None:
+            status_str = f"{val_score:.2f}/{test_score:.2f}"
+        else:
+            status_str = f"{val_score:.2f}"
+        return status_str, val_score, test_score, metric_display, feedback_decision, higher_is_better
+
+    # Check feedback stage (no benchmark result, use feedback decision directly)
+    if feedback_decision is not None:
+        return ("OK" if feedback_decision else "X"), None, None, None, feedback_decision, True
+
+    # Check running stage
+    running_files = list(loop_path.rglob("**/running/**/*.pkl"))
+    if running_files:
+        return "R", None, None, None, None, True
+
+    # Check coding stage
+    coding_files = list(loop_path.rglob("**/coding/**/*.pkl"))
+    if coding_files:
+        return "C", None, None, None, None, True
+
+    # Has directory but no recognized files
+    return "?", None, None, None, None, True
+
+
+def get_max_loops(job_path: Path) -> int:
+    """Get maximum number of loops across all tasks"""
+    max_loops = 0
+    for task_dir in job_path.iterdir():
+        if is_valid_task(task_dir):
+            loops = get_loop_dirs(task_dir)
+            max_loops = max(max_loops, len(loops))
+    return max_loops
+
+
+def get_job_summary_df(job_path: Path) -> tuple[pd.DataFrame, pd.DataFrame]:
+    """Generate summary DataFrame and decision DataFrame for all tasks in job
+
+    Each loop column shows "val/test" format when both scores are available.
+    Best columns show the best validation and test scores separately.
+
+    Returns:
+        (df, decisions_df): df is display data, decisions_df has same structure
+        but values are True/False/None for feedback decision
+    """
+    if not job_path.exists():
+        return pd.DataFrame(), pd.DataFrame()
+
+    tasks = [d for d in sorted(job_path.iterdir(), reverse=True) if is_valid_task(d)]
+    if not tasks:
+        return pd.DataFrame(), pd.DataFrame()
+
+    max_loops = get_max_loops(job_path)
+    if max_loops == 0:
+        max_loops = 10  # Default display columns
+
+    data = []
+    decisions_data = []
+    for task_path in tasks:
+        row = {"Task": task_path.name}
+        decision_row = {"Task": task_path.name}
+        best_val_score = None
+        best_test_score = None
+        best_metric = None
+        best_higher_is_better = True  # Default to higher is better
+
+        # Extract baseline scores (validation and test) from scenario
+        baseline_scores = extract_baseline_scores(task_path)
+        val_baseline = baseline_scores.get("validation")
+        test_baseline = baseline_scores.get("test")
+        if val_baseline and test_baseline:
+            row["Baseline"] = f"{val_baseline[1]:.2f}/{test_baseline[1]:.2f}"
+        elif val_baseline:
+            row["Baseline"] = f"{val_baseline[1]:.2f}"
+        else:
+            row["Baseline"] = "-"
+        decision_row["Baseline"] = None
+
+        for i in range(max_loops):
+            status, val_score, test_score, metric_name, feedback_decision, higher_is_better = get_loop_status(
+                task_path, i
+            )
+            row[f"L{i}"] = status
+            decision_row[f"L{i}"] = feedback_decision
+            if val_score is not None:
+                # Use higher_is_better to determine if this score is better
+                if best_val_score is None:
+                    best_val_score = val_score
+                    best_higher_is_better = higher_is_better
+                    best_metric = metric_name
+                elif (higher_is_better and val_score > best_val_score) or (
+                    not higher_is_better and val_score < best_val_score
+                ):
+                    best_val_score = val_score
+                    best_higher_is_better = higher_is_better
+                    best_metric = metric_name
+            if test_score is not None:
+                # Use same direction as validation score for consistency
+                if best_test_score is None:
+                    best_test_score = test_score
+                elif (best_higher_is_better and test_score > best_test_score) or (
+                    not best_higher_is_better and test_score < best_test_score
+                ):
+                    best_test_score = test_score
+
+        # Show best validation and test scores
+        if best_val_score is not None and best_test_score is not None:
+            row["Best"] = f"{best_val_score:.2f}/{best_test_score:.2f}"
+        elif best_val_score is not None:
+            row["Best"] = f"{best_val_score:.2f}"
+        else:
+            row["Best"] = "-"
+        row["Metric"] = best_metric if best_metric else "-"
+        decision_row["Metric"] = None
+        decision_row["Best"] = None
+        data.append(row)
+        decisions_data.append(decision_row)
+
+    # Ensure column order: Task, Metric, Baseline, L0, L1, ..., Best
+    df = pd.DataFrame(data)
+    decisions_df = pd.DataFrame(decisions_data)
+    if not df.empty:
+        loop_cols = [c for c in df.columns if c.startswith("L")]
+        cols = ["Task", "Metric", "Baseline"] + sorted(loop_cols, key=lambda x: int(x[1:])) + ["Best"]
+        df = df[cols]
+        decisions_df = decisions_df[cols]
+    return df, decisions_df
+
+
+def style_status_cell(val: str, decision: bool | None = None) -> str:
+    """Style cell based on status value and feedback decision
+
+    Args:
+        val: The cell value
+        decision: True=accepted (green), False=rejected (red), None=no feedback (gray)
+    """
+    if val == "-":
+        return "color: #888"
+    if val == "C":
+        return "color: #f0ad4e; font-weight: bold"  # Orange for coding
+    if val == "R":
+        return "color: #5bc0de; font-weight: bold"  # Blue for running
+    if val == "X":
+        return "color: #d9534f; font-weight: bold"  # Red for failed
+    if val == "OK":
+        return "color: #5cb85c; font-weight: bold"  # Green for success
+    if val == "?":
+        return "color: #888"
+
+    # Check if it's a numeric score (with optional "/" separator)
+    is_numeric = False
+    try:
+        float(val)
+        is_numeric = True
+    except ValueError:
+        if "/" in val:
+            parts = val.split("/")
+            try:
+                float(parts[0])
+                is_numeric = True
+            except ValueError:
+                pass
+
+    if is_numeric:
+        # Use decision for coloring (use == instead of is for numpy.bool_ compatibility)
+        if decision == True:
+            return "color: #5cb85c; font-weight: bold"  # Green for accepted
+        elif decision == False:
+            return "color: #d9534f; font-weight: bold"  # Red for rejected
+        else:
+            return "color: #888"  # Gray for no feedback
+
+    return ""
+
+
+def style_df_with_decisions(df: pd.DataFrame, decisions_df: pd.DataFrame) -> Styler:
+    """Apply styling to dataframe based on decision data
+
+    Args:
+        df: Display dataframe
+        decisions_df: DataFrame with same shape, containing True/False/None values
+    """
+
+    def apply_styles(row_idx: int, col: str) -> str:
+        val = df.iloc[row_idx][col]
+        decision = decisions_df.iloc[row_idx][col] if col in decisions_df.columns else None
+        return style_status_cell(str(val), decision)
+
+    # Build style matrix
+    styles = pd.DataFrame("", index=df.index, columns=df.columns)
+    for row_idx in range(len(df)):
+        for col in df.columns:
+            styles.iloc[row_idx][col] = apply_styles(row_idx, col)
+
+    return df.style.apply(lambda _: styles, axis=None)
+
+
+def render_job_summary(job_path: Path, is_root: bool = False) -> None:
+    """Render job summary UI"""
+    title = "Standalone Tasks" if is_root else f"Job: {job_path.name}"
+    st.subheader(title)
+
+    df, decisions_df = get_job_summary_df(job_path)
+    if df.empty:
+        st.warning("No valid tasks found in this job directory")
+        return
+
+    # Display legend
+    st.markdown(
+        "**Legend:** "
+        "<span style='color:#f0ad4e'>C</span>=Coding, "
+        "<span style='color:#5bc0de'>R</span>=Running, "
+        "<span style='color:#5cb85c'>Score</span>=Accepted, "
+        "<span style='color:#d9534f'>Score/X</span>=Rejected/Failed, "
+        "<span style='color:#888'>Score</span>=No feedback",
+        unsafe_allow_html=True,
+    )
+
+    # Style and display dataframe
+    styled_df = style_df_with_decisions(df, decisions_df)
+    st.dataframe(styled_df, use_container_width=True, hide_index=True)
+
+    # Summary stats
+    col1, col2, col3 = st.columns(3)
+    with col1:
+        st.metric("Tasks", len(df))
+    with col2:
+        # Count tasks with any score
+        tasks_with_score = df["Best"].apply(lambda x: x != "-").sum()
+        st.metric("With Score", tasks_with_score)
+    with col3:
+        # Count tasks with at least one improved loop (decision=True)
+        loop_cols = [c for c in decisions_df.columns if c.startswith("L")]
+        tasks_improved = decisions_df[loop_cols].apply(lambda row: any(v is True for v in row), axis=1).sum()
+        st.metric("Improved", tasks_improved)
+
+    # Detailed scores table
+    render_task_detail_selector(job_path)
+
+
+def extract_full_benchmark(loop_path: Path, split: str = "") -> dict | None:
+    """Extract full accuracy_summary from loop directory.
+
+    Args:
+        loop_path: Path to loop directory
+        split: Filter by split type ("validation", "test", or "" for any)
+
+    Returns:
+        accuracy_summary dict {dataset: {metric: value, ...}, ...} or None
+    """
+    for pkl_file in loop_path.rglob("**/benchmark_result*/**/*.pkl"):
+        try:
+            with open(pkl_file, "rb") as f:
+                content = pickle.load(f)
+            if isinstance(content, dict):
+                # Check split filter
+                content_split = content.get("split", "")
+                if split and content_split != split:
+                    continue
+
+                accuracy_summary = content.get("accuracy_summary", {})
+                if isinstance(accuracy_summary, dict) and accuracy_summary:
+                    return accuracy_summary
+        except Exception:
+            pass
+    return None
+
+
+def extract_baseline_full_benchmark(task_path: Path, split: str = "validation") -> dict | None:
+    """Extract full accuracy_summary from baseline scenario.
+
+    Args:
+        task_path: Path to task directory
+        split: "validation" or "test"
+
+    Returns:
+        accuracy_summary dict or None
+    """
+    scenario_dir = task_path / "scenario"
+    if not scenario_dir.exists():
+        return None
+
+    for pkl_file in scenario_dir.rglob("*.pkl"):
+        try:
+            with open(pkl_file, "rb") as f:
+                scenario = pickle.load(f)
+
+            if split == "validation":
+                baseline = getattr(scenario, "baseline_benchmark_score", None)
+            else:
+                baseline = getattr(scenario, "baseline_benchmark_score_test", None)
+
+            if baseline and isinstance(baseline, dict):
+                accuracy_summary = baseline.get("accuracy_summary", {})
+                if isinstance(accuracy_summary, dict) and accuracy_summary:
+                    return accuracy_summary
+        except Exception:
+            pass
+    return None
+
+
+def get_task_full_benchmark_df(task_path: Path, split: str) -> pd.DataFrame:
+    """Generate full benchmark table for a single task and split.
+
+    Returns DataFrame with columns: Dataset, Metric, Baseline, Loop_0, Loop_1, ...
+    Each row is a dataset-metric combination.
+    """
+    # Collect all sources (Baseline + Loops)
+    sources = ["Baseline"]
+    loop_dirs = sorted(
+        [d for d in task_path.iterdir() if d.is_dir() and d.name.startswith("Loop_")],
+        key=lambda x: int(x.name.split("_")[1]),
+    )
+    sources.extend([d.name for d in loop_dirs])
+
+    # Collect all accuracy_summaries
+    all_summaries = {}
+
+    # Baseline
+    baseline_summary = extract_baseline_full_benchmark(task_path, split)
+    if baseline_summary:
+        all_summaries["Baseline"] = baseline_summary
+
+    # Loops
+    for loop_dir in loop_dirs:
+        loop_summary = extract_full_benchmark(loop_dir, split)
+        if loop_summary:
+            all_summaries[loop_dir.name] = loop_summary
+
+    if not all_summaries:
+        return pd.DataFrame()
+
+    # Collect all dataset-metric combinations
+    all_keys = set()
+    for summary in all_summaries.values():
+        for dataset, metrics in summary.items():
+            if isinstance(metrics, dict):
+                for metric in metrics.keys():
+                    all_keys.add((dataset, metric))
+
+    # Sort keys for consistent display
+    all_keys = sorted(all_keys)
+
+    # Build table data
+    data = []
+    for dataset, metric in all_keys:
+        row = {"Dataset": dataset, "Metric": metric}
+        for source in sources:
+            summary = all_summaries.get(source, {})
+            metrics_dict = summary.get(dataset, {})
+            value = metrics_dict.get(metric) if isinstance(metrics_dict, dict) else None
+            if value is not None:
+                row[source] = f"{value:.2f}" if isinstance(value, float) else str(value)
+            else:
+                row[source] = "-"
+        data.append(row)
+
+    df = pd.DataFrame(data)
+    # Ensure column order
+    if not df.empty:
+        cols = ["Dataset", "Metric"] + [s for s in sources if s in df.columns]
+        df = df[cols]
+    return df
+
+
+def render_task_detail_selector(job_path: Path) -> None:
+    """Render task selector dropdown and full benchmark tables."""
+    tasks = [d for d in sorted(job_path.iterdir(), reverse=True) if is_valid_task(d)]
+    if not tasks:
+        return
+
+    st.markdown("---")
+    st.subheader("Detailed Benchmark Scores")
+
+    # Task selector dropdown
+    task_names = [t.name for t in tasks]
+    selected_task = st.selectbox("Select Task", options=task_names, index=0, key="task_detail_selector")
+
+    if selected_task:
+        task_path = job_path / selected_task
+
+        # Display Validation and Test tables side by side
+        col1, col2 = st.columns(2)
+
+        with col1:
+            st.markdown("**Validation**")
+            df_val = get_task_full_benchmark_df(task_path, "validation")
+            if not df_val.empty:
+                st.dataframe(df_val, use_container_width=True, hide_index=True)
+            else:
+                st.info("No validation scores")
+
+        with col2:
+            st.markdown("**Test**")
+            df_test = get_task_full_benchmark_df(task_path, "test")
+            if not df_test.empty:
+                st.dataframe(df_test, use_container_width=True, hide_index=True)
+            else:
+                st.info("No test scores")

rdagent/app/finetune/share/eval.py

@@ -0,0 +1,52 @@
+from pathlib import Path
+
+from rdagent.components.coder.CoSTEER.evaluators import (
+    CoSTEEREvaluator,
+    CoSTEERSingleFeedback,
+)
+from rdagent.core.experiment import FBWorkspace, Task
+from rdagent.core.scenario import Scenario
+from rdagent.utils.agent.tpl import T
+from rdagent.utils.agent.workflow import build_cls_from_json_with_retry
+
+
+class PrevModelLoadEvaluator(CoSTEEREvaluator):
+    """This evaluator checks whether the code actually loads a model from `prev_model`."""
+
+    def __init__(self, scen: Scenario):
+        super().__init__(scen)
+
+    def evaluate(
+        self, target_task: Task, implementation: FBWorkspace, gt_implementation: FBWorkspace, *args, **kwargs
+    ) -> CoSTEERSingleFeedback:
+        data_source_path = T("scenarios.data_science.share:scen.input_path").r()
+        prev_model_dir = Path(data_source_path) / "prev_model"
+
+        # 1) Inspect the code itself for references to prev_model loading
+        code_str = implementation.file_dict["main.py"]
+        code_contain_prev = "prev_model" in code_str
+        print(f"Code references prev_model: {code_contain_prev}")
+        if not code_contain_prev:
+            err = (
+                "No evidence found that your code loads a model from `prev_model`. "
+                "Please check that you are calling the correct load function "
+                f"and pointing it to the `{prev_model_dir}` directory."
+            )
+            return CoSTEERSingleFeedback(
+                execution=err,
+                return_checking=err,
+                code=err,
+                final_decision=False,
+            )
+
+        system_prompt = T(".prompts:prev_model_eval.system").r()
+        user_prompt = T(".prompts:prev_model_eval.user").r(
+            code=implementation.all_codes,
+        )
+
+        csfb = build_cls_from_json_with_retry(
+            CoSTEERSingleFeedback,
+            system_prompt=system_prompt,
+            user_prompt=user_prompt,
+        )
+        return csfb

rdagent/app/finetune/share/prompts.yaml

@@ -0,0 +1,23 @@
+prev_model_eval:
+  system: |-
+    You are a data scientist tasked with evaluating code generation. 
+
+    You will receive the following information:
+    - The implemented code
+
+    Focus on these aspects:
+    - Check if the code load the model in the "prev_model/" subfolder.
+
+    Please respond with your feedback in the following JSON format and order
+    ```json
+    {
+        "execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information. ."
+        "return_checking": "Detect whether the model is loaded from 'prev_model/' subfolder and finetune is prepared based on prev model.",
+        "code": "The code has explicity load the model from 'prev_model/' subfolder and prepares finetune based on prev model.",
+        "final_decision": <true or false in boolean type; only return true when ensuring that the code loads the model from 'prev_model/' subfolder and prepares finetune based on prev model.>
+    }
+    ```
+
+  user: |-
+    ------------ The implemented code ------------ 
+    {{code}}

rdagent/app/kaggle/conf.py

@@ -75,5 +75,22 @@ class KaggleBasePropSetting(ExtendedBaseSettings):
     mini_case: bool = False
     """Enable mini-case study for experiments"""
 
+    time_ratio_limit_to_enable_hyperparameter_tuning: float = 1
+    """
+    Runner time ratio limit to enable hyperparameter tuning, if not change, hyperparameter tuning is always enabled in the first evolution.
+    """
+
+    res_time_ratio_limit_to_enable_hyperparameter_tuning: float = 1
+    """
+    Overall rest time ratio limit to enable hyperparameter tuning, if not change, hyperparameter tuning is always enabled in the first evolution.
+    `1` indicate we enable hyperparameter tuning when we have 100% residual time. (so hyperparameter tuning is always enabled)
+    """
+
+    only_first_loop_enable_hyperparameter_tuning: bool = True
+    """Enable hyperparameter tuning feedback only in the first loop of evaluation."""
+
+    only_enable_tuning_in_merge: bool = False
+    """Enable hyperparameter tuning only in the merge stage"""
+
 
 KAGGLE_IMPLEMENT_SETTING = KaggleBasePropSetting()

rdagent/app/qlib_rd_loop/conf.py

@@ -1,3 +1,5 @@
+from typing import Optional
+
 from pydantic_settings import SettingsConfigDict
 
 from rdagent.components.workflow.conf import BasePropSetting
@@ -28,6 +30,24 @@ class ModelBasePropSetting(BasePropSetting):
     evolving_n: int = 10
     """Number of evolutions"""
 
+    train_start: str = "2008-01-01"
+    """Start date of the training segment"""
+
+    train_end: str = "2014-12-31"
+    """End date of the training segment"""
+
+    valid_start: str = "2015-01-01"
+    """Start date of the validation segment"""
+
+    valid_end: str = "2016-12-31"
+    """End date of the validation segment"""
+
+    test_start: str = "2017-01-01"
+    """Start date of the test / backtest segment"""
+
+    test_end: Optional[str] = "2020-08-01"
+    """End date of the test / backtest segment"""
+
 
 class FactorBasePropSetting(BasePropSetting):
     model_config = SettingsConfigDict(env_prefix="QLIB_FACTOR_", protected_namespaces=())
@@ -54,6 +74,24 @@ class FactorBasePropSetting(BasePropSetting):
     evolving_n: int = 10
     """Number of evolutions"""
 
+    train_start: str = "2008-01-01"
+    """Start date of the training segment"""
+
+    train_end: str = "2014-12-31"
+    """End date of the training segment"""
+
+    valid_start: str = "2015-01-01"
+    """Start date of the validation segment"""
+
+    valid_end: str = "2016-12-31"
+    """End date of the validation segment"""
+
+    test_start: str = "2017-01-01"
+    """Start date of the test / backtest segment"""
+
+    test_end: Optional[str] = "2020-08-01"
+    """End date of the test / backtest segment"""
+
 
 class FactorFromReportPropSetting(FactorBasePropSetting):
     # 1) override the scen attribute
@@ -64,10 +102,10 @@ class FactorFromReportPropSetting(FactorBasePropSetting):
     report_result_json_file_path: str = "git_ignore_folder/report_list.json"
     """Path to the JSON file listing research reports for factor extraction"""
 
-    max_factors_per_exp: int = 10000
+    max_factors_per_exp: int = 6
     """Maximum number of factors implemented per experiment"""
 
-    report_limit: int = 10000
+    report_limit: int = 20
     """Maximum number of reports to process"""
 
 
@@ -113,6 +151,24 @@ class QuantBasePropSetting(BasePropSetting):
     action_selection: str = "bandit"
     """Action selection strategy: 'bandit' for bandit-based selection, 'llm' for LLM-based selection, 'random' for random selection"""
 
+    train_start: str = "2008-01-01"
+    """Start date of the training segment"""
+
+    train_end: str = "2014-12-31"
+    """End date of the training segment"""
+
+    valid_start: str = "2015-01-01"
+    """Start date of the validation segment"""
+
+    valid_end: str = "2016-12-31"
+    """End date of the validation segment"""
+
+    test_start: str = "2017-01-01"
+    """Start date of the test / backtest segment"""
+
+    test_end: Optional[str] = "2020-08-01"
+    """End date of the test / backtest segment"""
+
 
 FACTOR_PROP_SETTING = FactorBasePropSetting()
 FACTOR_FROM_REPORT_PROP_SETTING = FactorFromReportPropSetting()

rdagent/app/qlib_rd_loop/factor.py

@@ -3,18 +3,20 @@ Factor workflow with session control
 """
 
 import asyncio
-from typing import Any
+from pathlib import Path
+from typing import Any, Optional
 
 import fire
 
 from rdagent.app.qlib_rd_loop.conf import FACTOR_PROP_SETTING
 from rdagent.components.workflow.rd_loop import RDLoop
-from rdagent.core.exception import FactorEmptyError
+from rdagent.core.exception import CoderError, FactorEmptyError
 from rdagent.log import rdagent_logger as logger
 
 
 class FactorRDLoop(RDLoop):
-    skip_loop_error = (FactorEmptyError,)
+    skip_loop_error = (FactorEmptyError, CoderError)
+    skip_loop_error_stepname = "feedback"
 
     def running(self, prev_out: dict[str, Any]):
         exp = self.runner.develop(prev_out["coding"])
@@ -25,7 +27,16 @@ class FactorRDLoop(RDLoop):
         return exp
 
 
-def main(path=None, step_n=None, loop_n=None, all_duration=None, checkout=True):
+def main(
+    path: Optional[str] = None,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    all_duration: str | None = None,
+    checkout: bool = True,
+    checkout_path: Optional[str] = None,
+    base_features_path: Optional[str] = None,
+    **kwargs,
+):
     """
     Auto R&D Evolving loop for fintech factors.
 
@@ -36,11 +47,19 @@ def main(path=None, step_n=None, loop_n=None, all_duration=None, checkout=True):
         dotenv run -- python rdagent/app/qlib_rd_loop/factor.py $LOG_PATH/__session__/1/0_propose  --step_n 1   # `step_n` is a optional paramter
 
     """
+    if not checkout_path is None:
+        checkout = Path(checkout_path)
+
     if path is None:
-        model_loop = FactorRDLoop(FACTOR_PROP_SETTING)
+        factor_loop = FactorRDLoop(FACTOR_PROP_SETTING)
     else:
-        model_loop = FactorRDLoop.load(path, checkout=checkout)
-    asyncio.run(model_loop.run(step_n=step_n, loop_n=loop_n, all_duration=all_duration))
+        factor_loop = FactorRDLoop.load(path, checkout=checkout)
+
+    factor_loop._init_base_features(base_features_path)
+    if "user_interaction_queues" in kwargs and kwargs["user_interaction_queues"] is not None:
+        factor_loop._set_interactor(*kwargs["user_interaction_queues"])
+        factor_loop._interact_init_params()
+    asyncio.run(factor_loop.run(step_n=step_n, loop_n=loop_n, all_duration=all_duration))
 
 
 if __name__ == "__main__":

rdagent/app/qlib_rd_loop/factor_from_report.py

@@ -12,7 +12,7 @@ from rdagent.components.document_reader.document_reader import (
     load_and_process_pdfs_by_langchain,
 )
 from rdagent.core.conf import RD_AGENT_SETTINGS
-from rdagent.core.proposal import Hypothesis
+from rdagent.core.proposal import Hypothesis, HypothesisFeedback
 from rdagent.log import rdagent_logger as logger
 from rdagent.oai.llm_utils import APIBackend
 from rdagent.scenarios.qlib.experiment.factor_experiment import QlibFactorExperiment
@@ -126,6 +126,9 @@ class FactorReportLoop(FactorRDLoop, metaclass=LoopMeta):
                 ]
                 exp.sub_workspace_list = exp.sub_workspace_list[: FACTOR_FROM_REPORT_PROP_SETTING.max_factors_per_exp]
                 exp.sub_tasks = exp.sub_tasks[: FACTOR_FROM_REPORT_PROP_SETTING.max_factors_per_exp]
+                exp.base_features = self.plan["features"]
+                if exp.based_experiments:
+                    exp.based_experiments[-1].base_features = self.plan["features"]
                 logger.log_object(exp.hypothesis, tag="hypothesis generation")
                 logger.log_object(exp.sub_tasks, tag="experiment generation")
                 return exp

rdagent/app/qlib_rd_loop/model.py

@@ -15,7 +15,15 @@ class ModelRDLoop(RDLoop):
     skip_loop_error = (ModelEmptyError,)
 
 
-def main(path=None, step_n=None, loop_n=None, all_duration=None, checkout=True):
+def main(
+    path=None,
+    step_n: int | None = None,
+    loop_n: int | None = None,
+    all_duration: str | None = None,
+    checkout: bool = True,
+    base_features_path: str | None = None,
+    **kwargs,
+):
     """
     Auto R&D Evolving loop for fintech models
 
@@ -30,6 +38,10 @@ def main(path=None, step_n=None, loop_n=None, all_duration=None, checkout=True):
         model_loop = ModelRDLoop(MODEL_PROP_SETTING)
     else:
         model_loop = ModelRDLoop.load(path, checkout=checkout)
+    model_loop._init_base_features(base_features_path)
+    if "user_interaction_queues" in kwargs and kwargs["user_interaction_queues"] is not None:
+        model_loop._set_interactor(*kwargs["user_interaction_queues"])
+        model_loop._interact_init_params()
     asyncio.run(model_loop.run(step_n=step_n, loop_n=loop_n, all_duration=all_duration))
 
 

rdagent/app/qlib_rd_loop/prompts.yaml

@@ -1,15 +1,55 @@
 hypothesis_generation:
   system: |-
-    You are an expert in financial analysis. Your task is to generate a well-reasoned hypothesis based on the provided financial factors and report content.
-    Please ensure your response is in JSON format as shown below:
+    You are an expert quantitative researcher specialized in FX (foreign exchange) trading,
+    specifically EURUSD intraday strategies on 15-minute bars.
+
+    EURUSD domain knowledge you must apply:
+    - London session (08:00-12:00 UTC): highest volatility, trending behavior — favor momentum strategies
+    - NY session (13:00-17:00 UTC): second volatility peak, also trending
+    - Asian session (00:00-07:00 UTC): low volatility, mean-reverting behavior
+    - London/NY overlap (13:00-17:00 UTC): strongest directional moves of the day
+    - Weekend gap risk: avoid holding positions after Friday 20:00 UTC
+    - Spread cost: ~1.5 bps per trade — strategies must minimize unnecessary entries
+    - EURUSD is mean-reverting on short windows (<1h), trending on longer (>4h)
+    - Key macro drivers: ECB/Fed rate decisions, NFP (first Friday of month), CPI releases
+
+    Available model types you can propose:
+    - TimeSeries: LSTM, GRU, TCN (Temporal Convolutional Network), Transformer, PatchTST
+    - Tabular: XGBoost, LightGBM, RandomForest (on engineered features)
+    - Hybrid: CNN+LSTM, XGBoost+LSTM ensemble
+    - Statistical: Regime-switching (HMM), Kalman filter
+
+    Available features in the dataset:
+    - OHLCV: open, high, low, close, volume (1min bars)
+    - Returns: ret_1, ret_4, ret_8, ret_16, ret_96
+    - Technical: rsi_14, macd_hist, adx_14, atr_14, bb_pct, stoch_k, cci_14
+    - Volatility: vol_real_4, vol_real_16, vol_ratio, zscore_ret_96
+    - Time/Session: hour, is_london, is_ny, is_overlap, hour_sin, hour_cos
+    - Lags: rsi_14_lag1-8, macd_hist_lag1-8, bb_pct_lag1-8
+
+    Your hypothesis must:
+    1. Specify which session(s) the strategy targets
+    2. Name which model type to use and why it fits EURUSD
+    3. Include a session filter (is_london / is_ny)
+    4. Include a spread filter (only trade when expected |return| > 0.0003)
+    5. Specify target: classification (fwd_sign_4) or regression (fwd_ret_4)
+
+    Please ensure your response is in JSON format:
     {
-      "hypothesis": "A clear and concise hypothesis based on the provided information.",
-      "reason": "A detailed explanation supporting the generated hypothesis.",
+      "hypothesis": "A clear and concise trading hypothesis for EURUSD 1min.",
+      "reason": "Detailed explanation including session, model choice, and expected edge.",
+      "model_type": "One of: TimeSeries / Tabular / XGBoost",
+      "target_session": "london / ny / asian / all",
+      "expected_arr_range": "e.g. 8-12%"
     }
 
   user: |-
-    The following are the financial factors and their descriptions:
+    Previously tried approaches and their results:
     {{ factor_descriptions }}
 
-    The report content is as follows:
-    {{ report_content }}
+    Additional context:
+    {{ report_content }}
+
+    Generate a NEW hypothesis that is meaningfully different from what has been tried.
+    Focus on approaches that have NOT been tested yet.
+    Target: beat current best ARR of 9.62%.

rdagent/app/qlib_rd_loop/quant.py

@@ -15,6 +15,7 @@ from rdagent.core.developer import Developer
 from rdagent.core.exception import FactorEmptyError, ModelEmptyError
 from rdagent.core.proposal import (
     Experiment2Feedback,
+    ExperimentPlan,
     Hypothesis2Experiment,
     HypothesisFeedback,
     HypothesisGen,
@@ -23,6 +24,7 @@ from rdagent.core.scenario import Scenario
 from rdagent.core.utils import import_class
 from rdagent.log import rdagent_logger as logger
 from rdagent.scenarios.qlib.proposal.quant_proposal import QuantTrace
+from rdagent.utils.qlib import ALPHA20
 
 
 class QuantRDLoop(RDLoop):
@@ -62,6 +64,10 @@ class QuantRDLoop(RDLoop):
         self.model_summarizer: Experiment2Feedback = import_class(PROP_SETTING.model_summarizer)(scen)
         logger.log_object(self.model_summarizer, tag="model summarizer")
 
+        self.plan: ExperimentPlan = {
+            "features": ALPHA20,
+            "feature_codes": {},
+        }  # for user interaction
         self.trace = QuantTrace(scen=scen)
         super(RDLoop, self).__init__()
 
@@ -75,6 +81,11 @@ class QuantRDLoop(RDLoop):
                 else:
                     exp = self.model_hypothesis2experiment.convert(hypo, self.trace)
                 logger.log_object(exp.sub_tasks, tag="experiment generation")
+                exp.base_features = self.plan["features"]
+                exp.base_feature_codes = self.plan["feature_codes"]
+                if exp.based_experiments:
+                    exp.based_experiments[-1].base_features = self.plan["features"]
+                    exp.based_experiments[-1].base_feature_codes = self.plan["feature_codes"]
                 return {"propose": hypo, "exp_gen": exp}
             await asyncio.sleep(1)
 
@@ -107,18 +118,25 @@ class QuantRDLoop(RDLoop):
                 reason="",
                 decision=False,
             )
-            logger.log_object(feedback, tag="feedback")
-            self.trace.hist.append((prev_out["direct_exp_gen"]["exp_gen"], feedback))
         else:
             if prev_out["direct_exp_gen"]["propose"].action == "factor":
                 feedback = self.factor_summarizer.generate_feedback(prev_out["running"], self.trace)
             elif prev_out["direct_exp_gen"]["propose"].action == "model":
                 feedback = self.model_summarizer.generate_feedback(prev_out["running"], self.trace)
-            logger.log_object(feedback, tag="feedback")
-            self.trace.hist.append((prev_out["running"], feedback))
+        feedback = self._interact_feedback(feedback)
+        logger.log_object(feedback, tag="feedback")
+        return feedback
 
 
-def main(path=None, step_n=None, loop_n=None, all_duration=None, checkout=True):
+def main(
+    path=None,
+    step_n: int | None = None,
+    loop_n: int | None = None,
+    all_duration: str | None = None,
+    checkout: bool = True,
+    base_features_path: str | None = None,
+    **kwargs,
+):
     """
     Auto R&D Evolving loop for fintech factors.
     You can continue running session by
@@ -129,6 +147,10 @@ def main(path=None, step_n=None, loop_n=None, all_duration=None, checkout=True):
         quant_loop = QuantRDLoop(QUANT_PROP_SETTING)
     else:
         quant_loop = QuantRDLoop.load(path, checkout=checkout)
+    quant_loop._init_base_features(base_features_path)
+    if "user_interaction_queues" in kwargs and kwargs["user_interaction_queues"] is not None:
+        quant_loop._set_interactor(*kwargs["user_interaction_queues"])
+        quant_loop._interact_init_params()
 
     asyncio.run(quant_loop.run(step_n=step_n, loop_n=loop_n, all_duration=all_duration))
 

rdagent/app/rl/conf.py

@@ -0,0 +1,41 @@
+from pathlib import Path
+
+from pydantic_settings import SettingsConfigDict
+
+from rdagent.core.conf import ExtendedBaseSettings
+
+
+class RLPostTrainingPropSetting(ExtendedBaseSettings):
+    """RL Post-training dedicated property settings.
+
+    Use RL_ env prefix for overrides.
+    """
+
+    model_config = SettingsConfigDict(env_prefix="RL_", protected_namespaces=())
+
+    # Main Components
+    scen: str = "rdagent.scenarios.rl.scen.scenario.RLPostTrainingScen"
+    hypothesis_gen: str = "rdagent.scenarios.rl.proposal.proposal.RLPostTrainingExpGen"
+    coder: str = "rdagent.components.coder.rl.RLCoSTEER"
+    runner: str = "rdagent.scenarios.rl.train.runner.RLPostTrainingRunner"
+    summarizer: str = "rdagent.scenarios.rl.dev.feedback.RLExperiment2Feedback"
+
+    # Resource paths (unified directory management, similar to SFT)
+    file_path: Path = Path.cwd() / "git_ignore_folder" / "rl_files"
+    """RL resource root directory. Contains datasets/ and models/ subdirectories.
+    Can be overridden via RL_FILE_PATH environment variable."""
+
+    # Core config
+    base_model: str | None = None
+    """Model name (e.g., 'Qwen2.5-Coder-0.5B-Instruct'). Docker path: /models/{base_model}"""
+
+    benchmark: str | None = None
+    """Benchmark/dataset name (e.g., 'gsm8k'). Docker path: /data/{benchmark}"""
+
+    # Benchmark evaluation
+    benchmark_timeout: int = 0
+    """Benchmark evaluation timeout in seconds. 0 means no timeout."""
+
+
+# Global setting instance
+RL_RD_SETTING = RLPostTrainingPropSetting()

rdagent/app/rl/loop.py

@@ -0,0 +1,64 @@
+"""
+RL Post-training Entry Point
+"""
+
+import asyncio
+from typing import Optional
+
+import typer
+from typing_extensions import Annotated
+
+from rdagent.app.rl.conf import RL_RD_SETTING
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.rl.loop import RLPostTrainingRDLoop
+
+
+def main(
+    base_model: Annotated[Optional[str], typer.Option("--base-model", "-m")] = None,
+    benchmark: Annotated[Optional[str], typer.Option("--benchmark", "-b")] = None,
+    step_n: Optional[int] = None,
+    loop_n: Optional[int] = None,
+    timeout: Optional[str] = None,
+):
+    """
+    RL post-training entry point
+
+    Parameters
+    ----------
+    base_model : str
+        Model name (e.g., 'Qwen2.5-Coder-0.5B-Instruct')
+        Docker path: /models/{base_model}
+    benchmark : str
+        Benchmark/dataset name (e.g., 'gsm8k')
+        Docker path: /data/{benchmark}
+    step_n : int, optional
+        Number of steps to run; if None, runs all steps per loop
+    loop_n : int, optional
+        Number of loops to run; if None, runs indefinitely
+    timeout : str, optional
+        Maximum duration for the entire process
+
+    Examples
+    --------
+    .. code-block:: bash
+
+        export RL_MODELS_DIR=/path/to/models
+        export RL_DATA_DIR=/path/to/data
+        python rdagent/app/rl/loop.py --base-model Qwen2.5-Coder-0.5B-Instruct --benchmark gsm8k
+    """
+    # Update config from CLI
+    if base_model:
+        RL_RD_SETTING.base_model = base_model
+    if benchmark:
+        RL_RD_SETTING.benchmark = benchmark
+
+    logger.info(f"Starting RL post-training: model={RL_RD_SETTING.base_model}, benchmark={RL_RD_SETTING.benchmark}")
+
+    # RDLoop 会自动根据 RL_RD_SETTING.scen 创建 Scenario
+    # Scenario.__init__() 中会自动运行 baseline 评测
+    loop = RLPostTrainingRDLoop(RL_RD_SETTING)
+    asyncio.run(loop.run(step_n=step_n, loop_n=loop_n, all_duration=timeout))
+
+
+if __name__ == "__main__":
+    typer.run(main)

rdagent/app/rl/ui/__init__.py

@@ -0,0 +1 @@
+"""RL Post-training UI"""

rdagent/app/rl/ui/app.py

@@ -0,0 +1,176 @@
+"""
+RL Post-training Timeline Viewer
+Hierarchical view: Session > Loop > Stage > Events
+
+Run:
+    streamlit run rdagent/app/rl/ui/app.py
+"""
+
+import os
+from pathlib import Path
+
+import streamlit as st
+from streamlit import session_state as state
+
+from rdagent.app.rl.ui.components import render_session, render_summary
+from rdagent.app.rl.ui.config import ALWAYS_VISIBLE_TYPES, OPTIONAL_TYPES
+from rdagent.app.rl.ui.data_loader import get_summary, get_valid_sessions, load_session
+from rdagent.app.rl.ui.rl_summary import render_job_summary
+
+DEFAULT_LOG_BASE = "log/"
+
+
+def _safe_resolve(user_input: str | None, safe_root: Path) -> Path:
+    """Resolve user path relative to safe_root; raise ValueError if it escapes."""
+    safe_root = safe_root.expanduser().resolve()
+    if not user_input:
+        return safe_root
+
+    if "\x00" in user_input:
+        raise ValueError("Invalid path: contains null byte")
+
+    try:
+        normalized = os.path.normpath(user_input)
+        if os.path.splitdrive(normalized)[0]:
+            raise ValueError("Absolute paths with drive letters are not allowed")
+        path_obj = Path(normalized).expanduser()
+        if path_obj.is_absolute():
+            raise ValueError("Absolute paths are not allowed")
+        candidate = (safe_root / path_obj).resolve(strict=False)
+        candidate.relative_to(safe_root)
+        return candidate
+    except (OSError, ValueError) as exc:
+        raise ValueError(f"Invalid path outside of allowed root: {user_input}") from exc
+
+
+def get_job_options(base_path: Path) -> list[str]:
+    """Scan directory and return job options list."""
+    options = []
+    has_root_tasks = False
+    job_dirs = []
+
+    if not base_path.exists():
+        return options
+
+    for d in base_path.iterdir():
+        if not d.is_dir():
+            continue
+        if (d / "__session__").exists():
+            has_root_tasks = True
+        else:
+            try:
+                if any((sub / "__session__").exists() for sub in d.iterdir() if sub.is_dir()):
+                    job_dirs.append(d.name)
+            except PermissionError:
+                pass
+
+    job_dirs.sort(reverse=True)
+    options.extend(job_dirs)
+    if has_root_tasks:
+        options.append(". (Current)")
+
+    return options
+
+
+def main():
+    st.set_page_config(layout="wide", page_title="RL Timeline", page_icon="🤖")
+
+    with st.sidebar:
+        view_mode = st.radio("View Mode", ["Job Summary", "Single Task"], horizontal=True)
+        st.divider()
+
+        default_log = os.environ.get("RL_LOG_PATH", DEFAULT_LOG_BASE)
+        safe_root = Path(default_log).expanduser().resolve()
+        job_folder = str(safe_root)
+        selected_types = ALWAYS_VISIBLE_TYPES.copy()
+        is_root_job = False
+
+        if view_mode == "Job Summary":
+            st.header("Job")
+            base_folder = st.text_input("Base Folder", value=default_log, key="base_folder_input")
+            try:
+                base_path = _safe_resolve(base_folder, safe_root)
+            except ValueError as e:
+                st.error(str(e))
+                return
+
+            job_options = get_job_options(base_path)
+            if job_options:
+                selected_job = st.selectbox("Select Job", job_options, key="job_select")
+                if selected_job.startswith("."):
+                    job_folder = str(base_path)
+                    is_root_job = True
+                else:
+                    job_folder = str(base_path / selected_job)
+                state.selected_job_folder = job_folder
+            else:
+                st.warning("No jobs found in this directory")
+                job_folder = str(base_path)
+
+            if st.button("Refresh", type="primary", key="refresh_job"):
+                st.rerun()
+        else:
+            st.header("Session")
+            default_path = getattr(state, "selected_job_folder", default_log)
+            log_folder = st.text_input("Log Folder", value=default_path)
+            try:
+                log_path = _safe_resolve(log_folder, safe_root)
+            except ValueError as e:
+                st.error(str(e))
+                return
+
+            sessions = get_valid_sessions(log_path)
+            if not sessions:
+                st.warning("No valid sessions found")
+                return
+
+            selected_session = st.selectbox("Session", sessions)
+
+            if st.button("Load", type="primary") or "session" not in state:
+                with st.spinner("Loading..."):
+                    state.session = load_session(log_path / selected_session)
+                    state.session_name = selected_session
+
+            st.divider()
+
+            st.subheader("Show More")
+            selected_types = ALWAYS_VISIBLE_TYPES.copy()
+            for event_type, (label, default) in OPTIONAL_TYPES.items():
+                if st.toggle(label, value=default, key=f"toggle_{event_type}"):
+                    selected_types.append(event_type)
+
+            st.divider()
+
+            if "session" in state:
+                summary = get_summary(state.session)
+                st.subheader("Summary")
+                st.metric("Loops", summary.get("loop_count", 0))
+                st.metric("LLM Calls", summary.get("llm_call_count", 0))
+                success = summary.get("docker_success", 0)
+                fail = summary.get("docker_fail", 0)
+                st.metric("Docker", f"{success}✓ / {fail}✗")
+
+    if view_mode == "Job Summary":
+        st.title("📊 RL Job Summary")
+        job_path = Path(job_folder).resolve()
+        if job_path.exists():
+            render_job_summary(job_path, is_root=is_root_job)
+        else:
+            st.warning(f"Job folder not found: {job_folder}")
+        return
+
+    st.title("🤖 RL Timeline Viewer")
+
+    if "session" not in state:
+        st.info("Select a session and click **Load** to view")
+        return
+
+    session = state.session
+    summary = get_summary(session)
+    render_summary(summary)
+    st.divider()
+    render_session(session, selected_types)
+
+
+if __name__ == "__main__":
+    main()

rdagent/app/rl/ui/components.py

@@ -0,0 +1,315 @@
+"""
+RL UI Components - Event Renderers
+Simplified version without EvoLoop
+"""
+
+from typing import Any
+
+import streamlit as st
+
+from rdagent.app.rl.ui.config import ICONS
+from rdagent.app.rl.ui.data_loader import Event, Loop, Session
+
+
+def format_duration(seconds: float | None) -> str:
+    if seconds is None:
+        return ""
+    if seconds < 60:
+        return f"{seconds:.1f}s"
+    minutes = int(seconds // 60)
+    secs = seconds % 60
+    return f"{minutes}m {secs:.0f}s"
+
+
+def render_session(session: Session, show_types: list[str]) -> None:
+    """Render full session"""
+    if session.init_events:
+        filtered = [e for e in session.init_events if e.type in show_types]
+        if filtered:
+            with st.expander("🚀 **Initialization**", expanded=False):
+                for event in filtered:
+                    render_event(event)
+
+    for loop_id in sorted(session.loops.keys()):
+        loop = session.loops[loop_id]
+        render_loop(loop, show_types)
+
+
+def render_loop(loop: Loop, show_types: list[str]) -> None:
+    """Render a single loop"""
+    # Get status indicators
+    docker_success = None
+    feedback_decision = None
+
+    for event in loop.running:
+        if event.type == "docker_exec" and event.success is not None:
+            docker_success = event.success
+
+    for event in loop.feedback:
+        if event.type == "feedback" and event.success is not None:
+            feedback_decision = event.success
+
+    # Build title
+    parts = []
+    if docker_success is not None:
+        parts.append("🐳✓" if docker_success else "🐳✗")
+    if feedback_decision is not None:
+        parts.append("✅" if feedback_decision else "❌")
+
+    result_str = " ".join(parts) if parts else ""
+
+    with st.expander(f"🔄 **Loop {loop.loop_id}** {result_str}", expanded=False):
+        # Proposal
+        if loop.proposal:
+            filtered = [e for e in loop.proposal if e.type in show_types]
+            if filtered:
+                st.markdown("#### 💡 Proposal")
+                for event in filtered:
+                    render_event(event)
+
+        # Coding
+        if loop.coding:
+            filtered = [e for e in loop.coding if e.type in show_types]
+            if filtered:
+                st.markdown("#### 💻 Coding")
+                for event in filtered:
+                    render_event(event)
+
+        # Running
+        if loop.running:
+            filtered = [e for e in loop.running if e.type in show_types]
+            if filtered:
+                st.markdown("#### 🏃 Running")
+                for event in filtered:
+                    render_event(event)
+
+        # Feedback
+        if loop.feedback:
+            filtered = [e for e in loop.feedback if e.type in show_types]
+            if filtered:
+                st.markdown("#### 📊 Feedback")
+                for event in filtered:
+                    render_event(event)
+
+
+def render_event(event: Event) -> None:
+    """Render a single event"""
+    icon = ICONS.get(event.type, "📌")
+    duration_str = f" ({format_duration(event.duration)})" if event.duration else ""
+
+    status = ""
+    if event.success is True:
+        status = "🟢 "
+    elif event.success is False:
+        status = "🔴 "
+
+    title = f"{event.time_str} {icon} {status}{event.title}{duration_str}"
+
+    renderers = {
+        "scenario": render_scenario,
+        "llm_call": render_llm_call,
+        "template": render_template,
+        "experiment": render_experiment,
+        "code": render_code,
+        "docker_exec": render_docker_exec,
+        "feedback": render_feedback,
+        "token": render_token,
+        "time": render_time_info,
+        "settings": render_settings,
+        "hypothesis": render_hypothesis,
+    }
+
+    renderer = renderers.get(event.type, render_generic)
+    with st.expander(title, expanded=False):
+        renderer(event.content)
+
+
+def render_scenario(content: Any) -> None:
+    if hasattr(content, "base_model"):
+        st.markdown(f"**Base Model:** `{content.base_model}`")
+    if hasattr(content, "benchmark"):
+        st.markdown(f"**Benchmark:** `{content.benchmark}`")
+    render_generic(content)
+
+
+def render_hypothesis(content: Any) -> None:
+    if hasattr(content, "hypothesis") and content.hypothesis:
+        st.markdown("**Hypothesis:**")
+        st.markdown(content.hypothesis)
+    if hasattr(content, "reason") and content.reason:
+        with st.expander("Reason", expanded=False):
+            st.markdown(content.reason)
+
+
+def render_settings(content: Any) -> None:
+    if isinstance(content, dict):
+        st.json(content)
+    else:
+        st.code(str(content), wrap_lines=True)
+
+
+def render_llm_call(content: Any) -> None:
+    if not isinstance(content, dict):
+        st.json(content) if content else st.info("No content")
+        return
+
+    if content.get("start") and content.get("end"):
+        duration = (content["end"] - content["start"]).total_seconds()
+        st.caption(f"Duration: {format_duration(duration)}")
+
+    system = content.get("system", "")
+    if system:
+        with st.expander("System Prompt", expanded=False):
+            st.code(system, language="text", line_numbers=True, wrap_lines=True)
+
+    user = content.get("user", "")
+    if user:
+        with st.expander("User Prompt", expanded=False):
+            st.code(user, language="text", line_numbers=True, wrap_lines=True)
+
+    resp = content.get("resp", "")
+    if resp:
+        st.markdown("**Response:**")
+        if resp.strip().startswith("{") or resp.strip().startswith("["):
+            st.code(resp, language="json", line_numbers=True, wrap_lines=True)
+        elif resp.strip().startswith("```"):
+            st.markdown(resp)
+        else:
+            st.code(resp, language="text", line_numbers=True, wrap_lines=True)
+
+
+def render_template(content: Any) -> None:
+    if not isinstance(content, dict):
+        st.json(content) if content else st.info("No content")
+        return
+
+    uri = content.get("uri", "")
+    st.caption(f"URI: `{uri}`")
+
+    context = content.get("context", {})
+    if context:
+        with st.expander("Context Variables", expanded=False):
+            st.json(context)
+
+    rendered = content.get("rendered", "")
+    if rendered:
+        with st.expander("Rendered", expanded=True):
+            st.code(rendered, language="text", line_numbers=True, wrap_lines=True)
+
+
+def render_experiment(content: Any) -> None:
+    if isinstance(content, list):
+        for i, task in enumerate(content):
+            if len(content) > 1:
+                st.markdown(f"**Task {i}**")
+            if hasattr(task, "description") and task.description:
+                st.markdown(task.description)
+    else:
+        render_generic(content)
+
+
+def render_code(content: Any) -> None:
+    if isinstance(content, list):
+        for ws in content:
+            if hasattr(ws, "file_dict") and ws.file_dict:
+                for filename, code in ws.file_dict.items():
+                    lang = "yaml" if filename.endswith((".yaml", ".yml")) else "python"
+                    with st.expander(filename, expanded=False):
+                        st.code(code, language=lang, line_numbers=True, wrap_lines=True)
+    elif hasattr(content, "file_dict") and content.file_dict:
+        for filename, code in content.file_dict.items():
+            lang = "yaml" if filename.endswith((".yaml", ".yml")) else "python"
+            with st.expander(filename, expanded=False):
+                st.code(code, language=lang, line_numbers=True, wrap_lines=True)
+    else:
+        render_generic(content)
+
+
+def render_docker_exec(content: Any) -> None:
+    if isinstance(content, dict):
+        exit_code = content.get("exit_code")
+        if exit_code is not None:
+            if exit_code == 0:
+                st.success(f"Exit code: {exit_code}")
+            else:
+                st.error(f"Exit code: {exit_code}")
+
+        stdout = content.get("stdout", "")
+        if stdout:
+            with st.expander("Output", expanded=True):
+                st.code(stdout, language="text", line_numbers=True, wrap_lines=True)
+    else:
+        render_generic(content)
+
+
+def render_feedback(content: Any) -> None:
+    # Handle benchmark result (dict)
+    if isinstance(content, dict):
+        if "accuracy" in content or "accuracy_summary" in content:
+            st.markdown("**Benchmark Result:**")
+            st.json(content)
+        else:
+            st.json(content)
+        return
+
+    # Handle HypothesisFeedback object
+    col1, col2 = st.columns(2)
+    with col1:
+        decision = getattr(content, "decision", None)
+        if decision is not None:
+            st.metric("Decision", "Accept" if decision else "Reject")
+
+    reason = getattr(content, "reason", None)
+    if reason:
+        with st.expander("Reason", expanded=True):
+            st.code(reason, language="text", line_numbers=True, wrap_lines=True)
+
+    code_change = getattr(content, "code_change_summary", None)
+    if code_change:
+        with st.expander("Code Change Summary", expanded=False):
+            st.markdown(code_change)
+
+
+def render_token(content: Any) -> None:
+    if isinstance(content, dict):
+        col1, col2, col3 = st.columns(3)
+        with col1:
+            st.metric("Prompt", content.get("prompt_tokens", 0))
+        with col2:
+            st.metric("Completion", content.get("completion_tokens", 0))
+        with col3:
+            st.metric("Total", content.get("total_tokens", 0))
+    else:
+        render_generic(content)
+
+
+def render_time_info(content: Any) -> None:
+    if isinstance(content, dict):
+        for k, v in content.items():
+            st.metric(k, f"{v:.1f}s" if isinstance(v, (int, float)) else str(v))
+    else:
+        render_generic(content)
+
+
+def render_generic(content: Any) -> None:
+    if hasattr(content, "__dict__"):
+        st.json(vars(content))
+    elif content:
+        st.json(content)
+    else:
+        st.info("No content")
+
+
+def render_summary(summary: dict) -> None:
+    col1, col2, col3, col4 = st.columns(4)
+    with col1:
+        st.metric("Loops", summary.get("loop_count", 0))
+    with col2:
+        st.metric("LLM Calls", summary.get("llm_call_count", 0))
+    with col3:
+        llm_time = summary.get("llm_total_time", 0)
+        st.metric("LLM Time", format_duration(llm_time))
+    with col4:
+        success = summary.get("docker_success", 0)
+        fail = summary.get("docker_fail", 0)
+        st.metric("Docker", f"{success}✓ / {fail}✗")

rdagent/app/rl/ui/config.py

@@ -0,0 +1,54 @@
+"""
+RL UI Configuration Constants
+"""
+
+from typing import Literal
+
+# Event type definition
+EventType = Literal[
+    "scenario",
+    "llm_call",
+    "template",
+    "experiment",
+    "code",
+    "docker_exec",
+    "feedback",
+    "token",
+    "time",
+    "settings",
+    "hypothesis",
+]
+
+# Event type icons
+ICONS = {
+    "scenario": "🎯",
+    "llm_call": "💬",
+    "template": "📋",
+    "experiment": "🧪",
+    "code": "📄",
+    "docker_exec": "🐳",
+    "feedback": "📊",
+    "token": "🔢",
+    "time": "⏱️",
+    "settings": "⚙️",
+    "hypothesis": "💡",
+}
+
+# Always visible event types
+ALWAYS_VISIBLE_TYPES = [
+    "scenario",
+    "hypothesis",
+    "llm_call",
+    "experiment",
+    "code",
+    "docker_exec",
+    "feedback",
+]
+
+# Optional event types with toggle config (label, default_enabled)
+OPTIONAL_TYPES = {
+    "template": ("📋 Template", False),
+    "token": ("🔢 Token", False),
+    "time": ("⏱️ Time", False),
+    "settings": ("⚙️ Settings", False),
+}

rdagent/app/rl/ui/data_loader.py

@@ -0,0 +1,309 @@
+"""
+RL UI Data Loader
+Load pkl logs and convert to hierarchical timeline structure
+Simplified version: no EvoLoop (RL doesn't have evolution loops)
+"""
+
+import pickle
+import re
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+import streamlit as st
+
+from rdagent.app.rl.ui.config import EventType
+from rdagent.log.storage import FileStorage
+
+
+@dataclass
+class Event:
+    """Timeline event"""
+
+    type: EventType
+    timestamp: datetime
+    tag: str
+    title: str
+    content: Any
+    loop_id: int | None = None
+    stage: str = ""
+    duration: float | None = None
+    success: bool | None = None
+
+    @property
+    def time_str(self) -> str:
+        return self.timestamp.strftime("%H:%M:%S")
+
+
+@dataclass
+class Loop:
+    """Main loop containing stages (no EvoLoop for RL)"""
+
+    loop_id: int
+    proposal: list[Event] = field(default_factory=list)  # hypothesis generation
+    coding: list[Event] = field(default_factory=list)  # code generation
+    running: list[Event] = field(default_factory=list)  # docker training + benchmark
+    feedback: list[Event] = field(default_factory=list)  # feedback
+
+
+@dataclass
+class Session:
+    """Session containing init events and loops"""
+
+    init_events: list[Event] = field(default_factory=list)
+    loops: dict[int, Loop] = field(default_factory=dict)
+
+
+def extract_loop_id(tag: str) -> int | None:
+    match = re.search(r"Loop_(\d+)", tag)
+    return int(match.group(1)) if match else None
+
+
+def extract_stage(tag: str) -> str:
+    if "proposal" in tag or "direct_exp_gen" in tag:
+        return "proposal"
+    if "coding" in tag:
+        return "coding"
+    if "running" in tag:
+        return "running"
+    if "feedback" in tag:
+        return "feedback"
+    return ""
+
+
+def get_valid_sessions(log_folder: Path) -> list[str]:
+    if not log_folder.exists():
+        return []
+    sessions = []
+    for d in log_folder.iterdir():
+        if d.is_dir() and d.joinpath("__session__").exists():
+            sessions.append(d.name)
+    return sorted(sessions, reverse=True)
+
+
+def parse_event(tag: str, content: Any, timestamp: datetime) -> Event | None:
+    loop_id = extract_loop_id(tag)
+    stage = extract_stage(tag)
+
+    # Scenario
+    if tag == "scenario":
+        return Event(type="scenario", timestamp=timestamp, tag=tag, title="Scenario", content=content)
+
+    # Settings
+    if "SETTINGS" in tag:
+        name = tag.replace("_SETTINGS", "").replace("SETTINGS", "")
+        return Event(type="settings", timestamp=timestamp, tag=tag, title=f"Settings: {name}", content=content)
+
+    # Hypothesis
+    if "hypothesis" in tag:
+        return Event(
+            type="hypothesis",
+            timestamp=timestamp,
+            tag=tag,
+            title="Hypothesis",
+            content=content,
+            loop_id=loop_id,
+            stage="proposal",
+        )
+
+    # LLM Call
+    if "debug_llm" in tag:
+        if isinstance(content, dict) and ("user" in content or "system" in content):
+            duration = None
+            if content.get("start") and content.get("end"):
+                duration = (content["end"] - content["start"]).total_seconds()
+            return Event(
+                type="llm_call",
+                timestamp=timestamp,
+                tag=tag,
+                title="LLM Call",
+                content=content,
+                loop_id=loop_id,
+                stage=stage,
+                duration=duration,
+            )
+
+    # Template
+    if "debug_tpl" in tag:
+        if isinstance(content, dict) and "uri" in content:
+            uri = content.get("uri", "")
+            tpl_name = uri.split(":")[-1] if ":" in uri else uri
+            return Event(
+                type="template",
+                timestamp=timestamp,
+                tag=tag,
+                title=f"Template: {tpl_name}",
+                content=content,
+                loop_id=loop_id,
+                stage=stage,
+            )
+
+    # Experiment/Coder result
+    if "coder result" in tag or "experiment generation" in tag:
+        return Event(
+            type="experiment",
+            timestamp=timestamp,
+            tag=tag,
+            title="Experiment",
+            content=content,
+            loop_id=loop_id,
+            stage=stage or "coding",
+        )
+
+    # Code
+    if "evolving code" in tag or "code" in tag.lower():
+        return Event(
+            type="code",
+            timestamp=timestamp,
+            tag=tag,
+            title="Code",
+            content=content,
+            loop_id=loop_id,
+            stage=stage or "coding",
+        )
+
+    # Docker run
+    if "docker_run" in tag:
+        exit_code = content.get("exit_code") if isinstance(content, dict) else None
+        success = exit_code == 0 if exit_code is not None else None
+        return Event(
+            type="docker_exec",
+            timestamp=timestamp,
+            tag=tag,
+            title=f"Docker Run {'✓' if success else '✗' if success is False else ''}",
+            content=content,
+            loop_id=loop_id,
+            stage="running",
+            success=success,
+        )
+
+    # Benchmark result
+    if "benchmark" in tag.lower():
+        return Event(
+            type="feedback",
+            timestamp=timestamp,
+            tag=tag,
+            title="Benchmark Result",
+            content=content,
+            loop_id=loop_id,
+            stage="running",
+        )
+
+    # Feedback
+    if "feedback" in tag:
+        decision = getattr(content, "decision", None)
+        return Event(
+            type="feedback",
+            timestamp=timestamp,
+            tag=tag,
+            title=f"Feedback: {'Accept' if decision else 'Reject'}",
+            content=content,
+            loop_id=loop_id,
+            stage="feedback",
+            success=decision,
+        )
+
+    # Token cost
+    if "token_cost" in tag:
+        if isinstance(content, dict):
+            total = content.get("total_tokens", 0)
+            return Event(
+                type="token",
+                timestamp=timestamp,
+                tag=tag,
+                title=f"Token: {total}",
+                content=content,
+                loop_id=loop_id,
+                stage=stage,
+            )
+
+    # Time info
+    if "time_info" in tag:
+        return Event(
+            type="time",
+            timestamp=timestamp,
+            tag=tag,
+            title="Time Info",
+            content=content,
+            loop_id=loop_id,
+            stage=stage,
+        )
+
+    return None
+
+
+@st.cache_data(ttl=300, hash_funcs={Path: str})
+def load_session(log_path: Path) -> Session:
+    """Load events into hierarchical session structure"""
+    session = Session()
+
+    # 手动遍历 pkl 文件，跳过无法加载的
+    events = []
+    pkl_files = sorted(log_path.rglob("*.pkl"))
+    for pkl_file in pkl_files:
+        if pkl_file.name == "debug_llm.pkl":
+            continue
+        try:
+            with open(pkl_file, "rb") as f:
+                content = pickle.load(f)
+            timestamp = datetime.strptime(pkl_file.stem, "%Y-%m-%d_%H-%M-%S-%f")
+            # 正确解析 tag：Loop_5/running/debug_tpl/2957404/xxx.pkl -> Loop_5.running.debug_tpl
+            tag = ".".join(pkl_file.relative_to(log_path).as_posix().replace("/", ".").split(".")[:-3])
+            event = parse_event(tag, content, timestamp)
+            if event:
+                events.append(event)
+        except (ModuleNotFoundError, ImportError, pickle.UnpicklingError, ValueError):
+            # 跳过无法加载的文件（不同 Python 版本或格式错误）
+            continue
+
+    events.sort(key=lambda e: e.timestamp)
+
+    for event in events:
+        if event.loop_id is None:
+            session.init_events.append(event)
+            continue
+
+        if event.loop_id not in session.loops:
+            session.loops[event.loop_id] = Loop(loop_id=event.loop_id)
+        loop = session.loops[event.loop_id]
+
+        if event.stage == "proposal":
+            loop.proposal.append(event)
+        elif event.stage == "coding":
+            loop.coding.append(event)
+        elif event.stage == "running":
+            loop.running.append(event)
+        elif event.stage == "feedback":
+            loop.feedback.append(event)
+        else:
+            loop.proposal.append(event)
+
+    return session
+
+
+def get_summary(session: Session) -> dict:
+    """Get summary statistics"""
+    llm_calls = []
+    docker_execs = []
+
+    for e in session.init_events:
+        if e.type == "llm_call":
+            llm_calls.append(e)
+        elif e.type == "docker_exec":
+            docker_execs.append(e)
+
+    for loop in session.loops.values():
+        for e in loop.proposal + loop.coding + loop.running + loop.feedback:
+            if e.type == "llm_call":
+                llm_calls.append(e)
+            elif e.type == "docker_exec":
+                docker_execs.append(e)
+
+    return {
+        "loop_count": len(session.loops),
+        "llm_call_count": len(llm_calls),
+        "llm_total_time": sum(e.duration or 0 for e in llm_calls),
+        "docker_success": sum(1 for e in docker_execs if e.success is True),
+        "docker_fail": sum(1 for e in docker_execs if e.success is False),
+    }

rdagent/app/rl/ui/rl_summary.py

@@ -0,0 +1,183 @@
+"""
+RL Job Summary View
+Display summary table for all tasks in a job directory
+"""
+
+import pickle
+from pathlib import Path
+
+import pandas as pd
+import streamlit as st
+
+
+def is_valid_task(task_path: Path) -> bool:
+    """Check if directory is a valid RL task (has __session__ subdirectory)"""
+    return task_path.is_dir() and (task_path / "__session__").exists()
+
+
+def get_loop_dirs(task_path: Path) -> list[Path]:
+    """Get sorted list of Loop directories"""
+    loops = [d for d in task_path.iterdir() if d.is_dir() and d.name.startswith("Loop_")]
+    return sorted(loops, key=lambda d: int(d.name.split("_")[1]))
+
+
+def get_loop_status(task_path: Path, loop_id: int) -> tuple[str, bool | None]:
+    """
+    Get loop status and feedback decision.
+    Returns: (status_str, feedback_decision)
+    Status: 'C'=Coding, 'R'=Running, 'X'=Failed, 'OK'=Success
+    """
+    loop_path = task_path / f"Loop_{loop_id}"
+    if not loop_path.exists():
+        return "-", None
+
+    # Check for feedback
+    feedback_decision = None
+    feedback_files = list(loop_path.rglob("**/feedback/**/*.pkl"))
+    for f in feedback_files:
+        try:
+            with open(f, "rb") as fp:
+                content = pickle.load(fp)
+            decision = getattr(content, "decision", None)
+            if decision is not None:
+                feedback_decision = decision
+                break
+        except Exception:
+            pass
+
+    if feedback_decision is not None:
+        return ("OK" if feedback_decision else "X"), feedback_decision
+
+    # Check running stage
+    running_files = list(loop_path.rglob("**/running/**/*.pkl"))
+    if running_files:
+        return "R", None
+
+    # Check coding stage
+    coding_files = list(loop_path.rglob("**/coding/**/*.pkl"))
+    if coding_files:
+        return "C", None
+
+    return "?", None
+
+
+def get_max_loops(job_path: Path) -> int:
+    """Get maximum number of loops across all tasks"""
+    max_loops = 0
+    for task_dir in job_path.iterdir():
+        if is_valid_task(task_dir):
+            loops = get_loop_dirs(task_dir)
+            max_loops = max(max_loops, len(loops))
+    return max_loops
+
+
+def get_job_summary_df(job_path: Path) -> tuple[pd.DataFrame, pd.DataFrame]:
+    """Generate summary DataFrame for all tasks in job"""
+    if not job_path.exists():
+        return pd.DataFrame(), pd.DataFrame()
+
+    tasks = [d for d in sorted(job_path.iterdir(), reverse=True) if is_valid_task(d)]
+    if not tasks:
+        return pd.DataFrame(), pd.DataFrame()
+
+    max_loops = get_max_loops(job_path)
+    if max_loops == 0:
+        max_loops = 10
+
+    data = []
+    decisions_data = []
+    for task_path in tasks:
+        row = {"Task": task_path.name}
+        decision_row = {"Task": task_path.name}
+        success_count = 0
+        fail_count = 0
+
+        for i in range(max_loops):
+            status, feedback_decision = get_loop_status(task_path, i)
+            row[f"L{i}"] = status
+            decision_row[f"L{i}"] = feedback_decision
+            if feedback_decision is True:
+                success_count += 1
+            elif feedback_decision is False:
+                fail_count += 1
+
+        row["Summary"] = f"{success_count}✓/{fail_count}✗"
+        decision_row["Summary"] = None
+        data.append(row)
+        decisions_data.append(decision_row)
+
+    df = pd.DataFrame(data)
+    decisions_df = pd.DataFrame(decisions_data)
+    if not df.empty:
+        loop_cols = [c for c in df.columns if c.startswith("L")]
+        cols = ["Task"] + sorted(loop_cols, key=lambda x: int(x[1:])) + ["Summary"]
+        df = df[cols]
+        decisions_df = decisions_df[cols]
+    return df, decisions_df
+
+
+def style_status_cell(val: str, decision: bool | None = None) -> str:
+    """Style cell based on status value"""
+    if val == "-":
+        return "color: #888"
+    if val == "C":
+        return "color: #f0ad4e; font-weight: bold"
+    if val == "R":
+        return "color: #5bc0de; font-weight: bold"
+    if val == "X":
+        return "color: #d9534f; font-weight: bold"
+    if val == "OK":
+        return "color: #5cb85c; font-weight: bold"
+    if val == "?":
+        return "color: #888"
+    return ""
+
+
+def style_df_with_decisions(df: pd.DataFrame, decisions_df: pd.DataFrame):
+    """Apply styling to dataframe"""
+
+    def apply_styles(row_idx: int, col: str) -> str:
+        val = df.iloc[row_idx][col]
+        decision = decisions_df.iloc[row_idx][col] if col in decisions_df.columns else None
+        return style_status_cell(str(val), decision)
+
+    styles = pd.DataFrame("", index=df.index, columns=df.columns)
+    for row_idx in range(len(df)):
+        for col in df.columns:
+            styles.iloc[row_idx][col] = apply_styles(row_idx, col)
+
+    return df.style.apply(lambda _: styles, axis=None)
+
+
+def render_job_summary(job_path: Path, is_root: bool = False) -> None:
+    """Render job summary UI"""
+    title = "Standalone Tasks" if is_root else f"Job: {job_path.name}"
+    st.subheader(title)
+
+    df, decisions_df = get_job_summary_df(job_path)
+    if df.empty:
+        st.warning("No valid tasks found in this job directory")
+        return
+
+    st.markdown(
+        "**Legend:** "
+        "<span style='color:#f0ad4e'>C</span>=Coding, "
+        "<span style='color:#5bc0de'>R</span>=Running, "
+        "<span style='color:#5cb85c'>OK</span>=Success, "
+        "<span style='color:#d9534f'>X</span>=Failed",
+        unsafe_allow_html=True,
+    )
+
+    styled_df = style_df_with_decisions(df, decisions_df)
+    st.dataframe(styled_df, use_container_width=True, hide_index=True)
+
+    col1, col2, col3 = st.columns(3)
+    with col1:
+        st.metric("Tasks", len(df))
+    with col2:
+        loop_cols = [c for c in decisions_df.columns if c.startswith("L")]
+        tasks_success = decisions_df[loop_cols].apply(lambda row: any(v is True for v in row), axis=1).sum()
+        st.metric("With Success", tasks_success)
+    with col3:
+        total_loops = sum(1 for _, row in decisions_df.iterrows() for c in loop_cols if row[c] is not None)
+        st.metric("Total Loops", total_loops)

rdagent/app/utils/health_check.py

@@ -1,12 +1,17 @@
+import os
 import socket
 
 import docker
+import fire
+import litellm
+from litellm import completion, embedding
+from litellm.utils import ModelResponse
 
 from rdagent.log import rdagent_logger as logger
 from rdagent.utils.env import cleanup_container
 
 
-def check_docker() -> None:
+def check_docker_status() -> None:
     container = None
     try:
         client = docker.from_env()
@@ -37,16 +42,127 @@ def check_and_list_free_ports(start_port=19899, max_ports=10) -> None:
             if not is_port_in_use(port):
                 free_ports.append(port)
         logger.warning(
-            f"Port 19899 is occupied, please replace it with an available port when running the `rdagent ui` command. Available ports: {free_ports}"
+            f"Port 19899 is occupied, please replace it with an available port when running the `rdagent ui/server_ui` command. Available ports: {free_ports}"
         )
     else:
-        logger.info(f"Port 19899 is not occupied, you can run the `rdagent ui` command")
+        logger.info(f"Port 19899 is not occupied, you can run the `rdagent ui/server_ui` command")
 
 
-def health_check():
+def test_chat(chat_model, chat_api_key, chat_api_base):
+    logger.info(f"🧪 Testing chat model: {chat_model}")
+    try:
+        if chat_api_base is None:
+            response: ModelResponse = completion(
+                model=chat_model,
+                api_key=chat_api_key,
+                messages=[
+                    {"role": "user", "content": "Hello!"},
+                ],
+            )
+        else:
+            response: ModelResponse = completion(
+                model=chat_model,
+                api_key=chat_api_key,
+                api_base=chat_api_base,
+                messages=[
+                    {"role": "user", "content": "Hello!"},
+                ],
+            )
+        logger.info(f"✅ Chat test passed.")
+        return True
+    except Exception as e:
+        logger.error(f"❌ Chat test failed: {e}")
+        return False
+
+
+def test_embedding(embedding_model, embedding_api_key, embedding_api_base):
+    logger.info(f"🧪 Testing embedding model: {embedding_model}")
+    try:
+        response = embedding(
+            model=embedding_model,
+            api_key=embedding_api_key,
+            api_base=embedding_api_base,
+            input="Hello world!",
+        )
+        logger.info("✅ Embedding test passed.")
+        return True
+    except Exception as e:
+        logger.error(f"❌ Embedding test failed: {e}")
+        return False
+
+
+def env_check():
+    if "BACKEND" not in os.environ:
+        logger.warning(
+            f"We did not find BACKEND in your configuration, please add it to your .env file. "
+            f"You can run a command like this: `dotenv set BACKEND rdagent.oai.backend.LiteLLMAPIBackend`"
+        )
+
+    if "DEEPSEEK_API_KEY" in os.environ:
+        chat_api_key = os.getenv("DEEPSEEK_API_KEY")
+        chat_model = os.getenv("CHAT_MODEL")
+        embedding_model = os.getenv("EMBEDDING_MODEL")
+        embedding_api_key = os.getenv("LITELLM_PROXY_API_KEY")
+        embedding_api_base = os.getenv("LITELLM_PROXY_API_BASE")
+        if "DEEPSEEK_API_BASE" in os.environ:
+            chat_api_base = os.getenv("DEEPSEEK_API_BASE")
+        elif "OPENAI_API_BASE" in os.environ:
+            chat_api_base = os.getenv("OPENAI_API_BASE")
+        else:
+            chat_api_base = None
+    elif "OPENAI_API_KEY" in os.environ:
+        chat_api_key = os.getenv("OPENAI_API_KEY")
+        chat_api_base = os.getenv("OPENAI_API_BASE")
+        chat_model = os.getenv("CHAT_MODEL")
+        embedding_model = os.getenv("EMBEDDING_MODEL")
+        embedding_api_key = chat_api_key
+        embedding_api_base = chat_api_base
+    else:
+        logger.error("No valid configuration was found, please check your .env file.")
+
+    logger.info("🚀 Starting test...\n")
+    result_embedding = test_embedding(
+        embedding_model=embedding_model, embedding_api_key=embedding_api_key, embedding_api_base=embedding_api_base
+    )
+    result_chat = test_chat(chat_model=chat_model, chat_api_key=chat_api_key, chat_api_base=chat_api_base)
+
+    if result_chat and result_embedding:
+        logger.info("✅ All tests completed.")
+    else:
+        logger.error(" One or more tests failed. Please check credentials or model support.")
+
+
+def health_check(
+    check_env: bool = True,
+    check_docker: bool = True,
+    check_ports: bool = True,
+):
     """
-    Check that docker is installed correctly,
-    and that the ports used in the sample README are not occupied.
+    Run the RD-Agent health check:
+    - Check if Docker is available
+    - Check that the default ports are not occupied
+    - (Optional) Check that the API Key and model are configured correctly.
+
+    Args:
+        check_env (bool): Whether to check API Key and model configuration.
+        check_docker (bool): Checks if Docker is installed and running.
+        check_ports (bool): Whether to check if the default port (19899) is occupied.
     """
-    check_docker()
-    check_and_list_free_ports()
+    check_any = False
+
+    if check_env:
+        check_any = True
+        env_check()
+    if check_docker:
+        check_any = True
+        check_docker_status()
+    if check_ports:
+        check_any = True
+        check_and_list_free_ports()
+
+    if not check_any:
+        logger.warning("⚠️ All health check items are disabled. Please enable at least one check.")
+
+
+if __name__ == "__main__":
+    fire.Fire(health_check)

rdagent/app/utils/info.py

@@ -5,6 +5,7 @@ from pathlib import Path
 
 import docker
 import requests
+from packaging.requirements import Requirement
 from setuptools_scm import get_version
 
 from rdagent.log import rdagent_logger as logger
@@ -69,10 +70,9 @@ def rdagent_info():
     ]
     package_version_list = []
     for package in package_list:
-        if package == "typer[all]":
-            package = "typer"
-        version = importlib.metadata.version(package)
-        package_version_list.append(f"{package}=={version}")
+        pkg = Requirement(package)
+        version = importlib.metadata.version(pkg.name)
+        package_version_list.append(f"{pkg.name}=={version}")
     logger.info(f"Package version: {package_version_list}")
     return None
 

rdagent/app/utils/ws.py

@@ -0,0 +1,54 @@
+from typing import Optional
+
+import typer
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.components.coder.data_science.conf import get_ds_env
+from rdagent.utils.agent.tpl import T
+
+app = typer.Typer(help="Run data-science environment commands.")
+
+
+@app.command()
+def run(competition: str, cmd: str, local_path: str = "./", mount_path: str | None = None):
+    """
+    Launch the data-science environment for a specific competition and run the
+    provided command.
+
+    Example:
+        1) start the container:
+        dotenv run -- python -m rdagent.app.utils.ws nomad2018-predict-transparent-conductors "sleep 3600" --local-path your_workspace
+
+        2) then run the following command to enter the latest container:
+        - docker exec -it `docker ps --filter 'status=running' -l --format '{{.Names}}'` bash
+        Or you can attach to the container by specifying the container name (find it in the run info)
+        - docker exec -it sweet_robinson bash
+
+    Arguments:
+        competition: The competition slug/folder name.
+        cmd: The shell command or script entry point to execute inside
+             the environment.
+    """
+    data_path = DS_RD_SETTING.local_data_path
+
+    data_path = (
+        f"{data_path}/{competition}" if DS_RD_SETTING.sample_data_by_LLM else f"{data_path}/sample/{competition}"
+    )
+    target_path = T("scenarios.data_science.share:scen.input_path").r()
+    extra_volumes = {data_path: target_path}
+
+    # Don't set time limitation and always disable cache
+    env = get_ds_env(
+        extra_volumes=extra_volumes,
+        running_timeout_period=None,
+        enable_cache=False,
+    )
+
+    if mount_path is not None:
+        env.conf.mount_path = mount_path
+
+    env.run(entry=cmd, local_path=local_path)
+
+
+if __name__ == "__main__":  # pragma: no cover
+    app()

rdagent/app/utils/ws_ft.py

@@ -0,0 +1,52 @@
+from typing import Optional
+
+import typer
+
+from rdagent.app.finetune.llm.conf import FT_RD_SETTING
+from rdagent.components.coder.finetune.conf import get_ft_env
+from rdagent.utils.agent.tpl import T
+
+app = typer.Typer(help="Run LLM fine-tuning environment commands.")
+
+
+@app.command()
+def run(
+    dataset: str,
+    model: str,
+    cmd: str,
+    local_path: str = "./",
+    mount_path: str | None = None,
+):
+    """
+    Launch the LLM fine-tuning environment for a specific dataset and model, then run the
+    provided command.
+
+    Example:
+        1) start the container:
+        dotenv run -- python -m rdagent.app.utils.ws_ft alpaca_gpt4_zh qwen2-7b "sleep 3600" --local-path your_workspace
+
+        2) then run the following command to enter the latest container:
+        - docker exec -it `docker ps --filter 'status=running' -l --format '{{.Names}}'` bash
+        Or you can attach to the container by specifying the container name (find it in the run info)
+        - docker exec -it sweet_robinson bash
+
+    Arguments:
+        dataset: The dataset name for fine-tuning.
+        model: The base model name for fine-tuning.
+        cmd: The shell command or script entry point to execute inside
+             the environment.
+    """
+    # Don't set time limitation and always disable cache
+    env = get_ft_env(
+        running_timeout_period=None,
+        enable_cache=False,
+    )
+
+    if mount_path is not None:
+        env.conf.mount_path = mount_path
+
+    env.run(entry=cmd, local_path=local_path)
+
+
+if __name__ == "__main__":  # pragma: no cover
+    app()

rdagent/components/agent/__init__.py

@@ -0,0 +1,3 @@
+"""
+Some agent that can be shared across different scenarios.
+"""

rdagent/components/agent/base.py

@@ -0,0 +1,79 @@
+from abc import abstractmethod
+
+import nest_asyncio
+from prefect import task
+from prefect.cache_policies import INPUTS
+from pydantic_ai import Agent
+from pydantic_ai.mcp import MCPServerStreamableHTTP
+
+from rdagent.oai.backend.pydantic_ai import get_agent_model
+
+
+class BaseAgent:
+
+    @abstractmethod
+    def __init__(self, system_prompt: str, toolsets: list[str]): ...
+
+    @abstractmethod
+    def query(self, query: str) -> str: ...
+
+
+class PAIAgent(BaseAgent):
+    """
+    Pydantic-AI agent with optional Prefect caching support
+    """
+
+    agent: Agent
+    enable_cache: bool
+
+    def __init__(
+        self,
+        system_prompt: str,
+        toolsets: list[str | MCPServerStreamableHTTP],
+        enable_cache: bool = False,
+    ):
+        """
+        Initialize Pydantic-AI agent
+
+        Parameters
+        ----------
+        system_prompt : str
+            System prompt for the agent
+        toolsets : list[str | MCPServerStreamableHTTP]
+            List of MCP server URLs or instances
+        enable_cache : bool
+            Enable persistent caching via Prefect. Requires Prefect server:
+            `prefect server start` then set PREFECT_API_URL in environment
+        """
+        toolsets = [(ts if isinstance(ts, MCPServerStreamableHTTP) else MCPServerStreamableHTTP(ts)) for ts in toolsets]
+        self.agent = Agent(get_agent_model(), system_prompt=system_prompt, toolsets=toolsets)
+        self.enable_cache = enable_cache
+
+        # Create cached query function if caching is enabled
+        if enable_cache:
+            self._cached_query = task(cache_policy=INPUTS, persist_result=True)(self._run_query)
+
+    def _run_query(self, query: str) -> str:
+        """
+        Internal query execution (no caching)
+        """
+        nest_asyncio.apply()  # NOTE: very important. Because pydantic-ai uses asyncio!
+        result = self.agent.run_sync(query)
+        return result.output
+
+    def query(self, query: str) -> str:
+        """
+        Run agent query with optional caching
+
+        Parameters
+        ----------
+        query : str
+
+        Returns
+        -------
+        str
+        """
+        if self.enable_cache:
+            return self._cached_query(query)
+        else:
+            return self._run_query(query)

rdagent/components/agent/context7/__init__.py

@@ -0,0 +1,59 @@
+from typing import Optional
+
+from pydantic_ai.mcp import MCPServerStreamableHTTP
+
+from rdagent.components.agent.base import PAIAgent
+from rdagent.components.agent.context7.conf import SETTINGS
+from rdagent.log import rdagent_logger as logger
+from rdagent.utils.agent.tpl import T
+
+
+class Agent(PAIAgent):
+    """
+    A specific agent for context7
+    """
+
+    def __init__(self):
+        toolsets = [MCPServerStreamableHTTP(SETTINGS.url, timeout=SETTINGS.timeout)]
+
+        super().__init__(
+            system_prompt=T(".prompts:system_prompt").r(),
+            toolsets=toolsets,
+            enable_cache=SETTINGS.enable_cache,
+        )
+
+    def _build_enhanced_query(self, error_message: str, full_code: Optional[str] = None) -> str:
+        """Build enhanced query using experimental prompt templates."""
+        # Build context information using template
+        context_info = ""
+        if full_code:
+            context_info = T(".prompts:code_context_template").r(full_code=full_code)
+
+        # Check for timm library special case (experimental optimization)
+        timm_trigger = error_message.lower().count("timm") >= 3
+        timm_trigger_text = ""
+        if timm_trigger:
+            timm_trigger_text = T(".prompts:timm_special_case").r()
+            logger.info("🎯 Timm special handling triggered", tag="context7")
+
+        # Construct enhanced query using experimental template
+        enhanced_query = T(".prompts:context7_enhanced_query_template").r(
+            error_message=error_message, context_info=context_info, timm_trigger_text=timm_trigger_text
+        )
+
+        return enhanced_query
+
+    def query(self, query: str) -> str:
+        """
+
+        Parameters
+        ----------
+        query : str
+            It should be something like error message.
+
+        Returns
+        -------
+        str
+        """
+        query = self._build_enhanced_query(error_message=query)
+        return super().query(query)

rdagent/components/agent/context7/conf.py

@@ -0,0 +1,31 @@
+"""
+The context7 is based on a modified version of the context7.
+
+You can follow the instructions to install it
+
+    mkdir -p ~/tmp/
+    cd ~/tmp/ && git clone https://github.com/Hoder-zyf/context7.git
+    cd ~/tmp/context7
+    npm install -g bun
+    bun i && bun run build
+    bun run dist/index.js --transport http --port 8124 # > bun.out 2>&1 &
+"""
+
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class Settings(BaseSettings):
+    """Project specific settings."""
+
+    url: str = "http://localhost:8124/mcp"
+    timeout: int = 120
+    enable_cache: bool = False
+    # set CONTEXT7_ENABLE_CACHE=true in .env to enable cache
+
+    model_config = SettingsConfigDict(
+        env_prefix="CONTEXT7_",
+        # extra="allow", # Does it allow extrasettings
+    )
+
+
+SETTINGS = Settings()