gavindiaz/NexQuant
1
0
Fork 0
mirror of https://github.com/NicolasBohn/NexQuant.git synced 2026-08-05 19:17:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: loader prompt & simplify YAML loading and update data loader specifications (#736)

Browse Source 
* refactor: Simplify YAML loading and update data loader specifications

* docs: Add comment explaining FunctionLoader usage in tpl.py

* lint

* lint
This commit is contained in:
you-n-g
2025-04-01 22:36:25 +08:00
committed by GitHub
parent 67ee1a90bd
commit 3d6ae62ad5
4 changed files with 68 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rdagent/components/coder/data_science/raw_data_loader/prompts.yaml
+4 -25
View File
@@ -51,33 +51,12 @@ spec:
- Specify the data source location (`/kaggle/input/`).
- Clearly define the structure and type of the output.
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
2. Precautions for Data Loading and Preprocessing:
- File Handling:
- Handle file encoding and delimiters appropriately.
- Combine or process multiple files if necessary.
- Avoid using the sample submission file to infer test indices. If a dedicated test index file is available, use that. If not, use the order in the test file as the test index.
- Data Preprocessing:
- Convert data types correctly (e.g., numeric, categorical, date parsing).
- Handle missing values appropriately.
- Optimize memory usage for large datasets using techniques like downcasting or reading data in chunks if necessary.
- Domain-Specific Handling:
- Apply competition-specific preprocessing steps as needed (e.g., text tokenization, image resizing).
3. Code Standards:
- DO NOT use progress bars (e.g., `tqdm`).
- DO NOT use the sample submission file to extract test index information.
- DO NOT exclude features inadvertently during this process.
4. Notes:
2. Notes:
- Update `DT` (data type) based on the specific competition dataset. This can include `pd.DataFrame`, `np.array`, `torch.Tensor`, etc.
- Never use sample submission as the test index, as it may not be the same as the test data. Use the test index file or test data source to get the test index.
- Only set the DT of variables without inferring the shape of these variables since you don't know the shape of the data.
5. Exploratory Data Analysis (EDA) [Required]:
- Before returning the data, you should always add an EDA part describing the data to help the following steps understand the data better.
- The EDA part should be drafted in plain text with certain format schema with no more than ten thousand characters.
- An evaluation agent will help to check whether the EDA part is added correctly.
Responsibilities and notes of an implemented data loader that aligns with the generated specification.
{% include "scenarios.data_science.share:component_spec.DataLoadSpec" %}
{% if latest_spec %}
6. Former Specification: