Files
NexQuant/test/utils
you-n-g 304d3fad2e feat: parallel loop running based on asyncio (#932)
* refactor: split workflow into pkg, add WorkflowTracker & wait_retry

* feat: add async LoopBase with parallel workers and step semaphores

* fix: replace pickle with dill and run blocking tasks via joblib wrapper

* feat: add log format settings, dynamic parallelism & pickle-based snapshot

* fix: default step semaphore to 1 and avoid subprocess when single worker

* merge bowen's changes

* merge tim's changes

* refactor: extract component task mapping, add conditional logger setup

* lint

* refactor: add type hints and safer remain_time metric logging in workflow

* lint

* fix: allow BadRequestError to be pickled via custom copyreg reducer

* fix: stop loop when LoopTerminationError is raised in LoopBase

* lint

* refactor: make log tag context-local using ContextVar for thread safety

* feat: add subproc_step flag and helper to decide subprocess execution

* fix: use ./cache path and normalize relative volume bind paths

* fix: reset loop_idx to 0 on loop restart/resume to ensure correct flow

* fix: avoid chmod on cache and input dirs in Env timeout wrapper

* fix: skip chmod on 'cache' and 'input' dirs using find -prune

* fix: restrict chmod to immediate mount dirs excluding cache/input

* fix: chmod cache and input dirs alongside their contents after entry run

* fix: guard chmod with directory checks for cache and input

* fix: prefix mount_path in chmod command for cache/input dirs

* fix: drop quotes from find exclude patterns to ensure chmod executes

* fix: skip chmod on cache/input directories to avoid warning spam

* feat: support string volume mappings and poll subprocess stdout/stderr

* support remove symbolic link

* test: use dynamic home path and code volume in LocalEnv local_simple

* fix: skip trace and progress update when loop step is withdrawn

* refactor: add clean_workspace util and non-destructive workspace backup

* fix: preserve symlinks when backing up workspace with copytree

* fix: prevent AttributeError when _pbar not yet initialized in LoopBase

* perf: replace shutil.copytree with rsync for faster workspace backup

* fix: cast log directory Path to str in tar command of data science loop

* fix: use portable 'cp -r -P' instead of rsync for workspace backup

* fix: add retry and logging to workspace backup for robustness

* refactor: extract backup_folder helper and reuse in DataScienceRDLoop

* fix: propagate backup errors & default _pbar getattr to avoid error

* fix the division by zero bug

* refactor: execute RD loops via asyncio.run and add necessary imports

* lint

* lint

* lint

---------

Co-authored-by: Xu <v-xuminrui@microsoft.com>
2025-06-12 11:44:14 +08:00
..
2024-07-11 08:49:37 +00:00
2024-11-20 17:01:18 +08:00
2024-09-06 17:18:52 +08:00

🐳 Run Docker & Qlib


📄 Description

This guide explains how to run the Qlib Docker test file located at test/utils/test_env.py in the RD-Agent repository.


🚀 Running Instructions

1. Install the required Python libraries

  • Ensure that the docker Python library is installed:
    pip install docker
    

2. Run the test script

  • Execute the test script to verify the Docker environment setup:
    python test/utils/test_env.py
    

Troubleshooting

  • PermissionError: [Errno 13] Permission denied.

    This error occurs when the current user does not have the necessary permissions to access the Docker socket. To resolve this issue, follow these steps:

  1. Add the current user to the docker group Docker requires root or docker group user permissions to access the Docker socket. Add the current user to the docker group:

    sudo usermod -aG docker $USER
    
  2. Refresh group changes To apply the group changes, log out and log back in, or use the following command:

    newgrp docker
    
  3. Verify Docker access Run the following command to ensure that Docker can be accessed:

    docker run hello-world
    
  4. Rerun the test script After completing these steps, rerun the test script:

    python test/utils/test_env.py
    

🛠️ Detailed Qlib Docker Function Framework

Here, we provide an overview of the specific functions within the Qlib Docker framework, their purposes, and examples of how to call them.

QTDockerEnv Class in env.py

The QTDockerEnv class is responsible for setting up and running Docker environments for Qlib experiments.

Methods:

  1. prepare()

    • Purpose: Prepares the Docker environment for running experiments. This includes building the Docker image if necessary.
    • Example:
      qtde = QTDockerEnv()
      qtde.prepare()
      
  2. run(local_path: str, entry: str) -> str

    • Purpose: Runs a specified entry point (e.g., a configuration file) in the prepared Docker environment.
    • Parameters:
      • local_path: Path to the local directory to mount into the Docker container.
      • entry: Command or entry point to run inside the Docker container.
    • Returns: The stdout output from the Docker container.
    • Example:
      result = qtde.run(local_path="/path/to/env_tpl", entry="qrun conf.yaml")
      

📊 Expected Output

Upon successful execution, the test script will produce analysis results of benchmark returns and various risk metrics. The expected output should be similar to:

'The following are analysis results of benchmark return (1 day).'
risk
mean               0.000477
std                0.012295
annualized_return  0.113561
information_ratio  0.598699
max_drawdown      -0.370479

'The following are analysis results of the excess return without cost (1 day).'
risk
mean               0.000530
std                0.005718
annualized_return  0.126029
information_ratio  1.428574
max_drawdown      -0.072310

'The following are analysis results of the excess return with cost (1 day).'
risk
mean               0.000339
std                0.005717
annualized_return  0.080654
information_ratio  0.914486
max_drawdown      -0.086083

'The following are analysis results of indicators (1 day).'
value
ffr    1.0
pa     0.0
pos    0.0

By following these steps and using the provided functions, you should be able to run the Qlib Docker tests and obtain the expected analysis results.