diff --git a/rdagent/components/coder/data_science/ensemble/eval_tests/ensemble_test.txt b/rdagent/components/coder/data_science/ensemble/eval_tests/ensemble_test.txt index 296c7ed1..90a1a7e9 100644 --- a/rdagent/components/coder/data_science/ensemble/eval_tests/ensemble_test.txt +++ b/rdagent/components/coder/data_science/ensemble/eval_tests/ensemble_test.txt @@ -95,5 +95,6 @@ assert model_set_in_scores == set({{model_names}}).union({"ensemble"}), ( f"The scores dataframe does not contain the correct model names as index.\ncorrect model names are: {{model_names}} + ['ensemble']\nscore_df is:\n{score_df}" ) assert score_df.index.is_unique, "The scores dataframe has duplicate model names." +assert len(score_df.columns) == 1, f"The scores dataframe should have exactly one column for the scores of the evaluation indicator, but has these columns: {score_df.columns.tolist()}" print("Ensemble test end.") diff --git a/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml b/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml index 6f98d590..2c88601d 100644 --- a/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml +++ b/rdagent/components/coder/data_science/raw_data_loader/prompts.yaml @@ -204,7 +204,7 @@ spec: - Verify that `val_label` is provided and matches the length of `val_preds_dict` predictions. - Handle empty or invalid inputs gracefully with appropriate error messages. - Metric Calculation and Storage: - - Calculate the metric for each model and ensemble strategy, and save the results in `scores.csv`, e.g.: + - Calculate the metric (mentioned in the evaluation section of the competition information) for each model and ensemble strategy, and save the results in `scores.csv`, e.g.: ```python scores = {} for model_name, val_pred in val_preds_dict.items(): @@ -259,7 +259,7 @@ spec: 3. Dataset Splitting - The dataset returned by `load_data` is not split into training and testing sets, so the dataset splitting should happen after calling `feat_eng`. - - Decide whether to use a **static train-test split** or **cross-validation**, based on what is most suitable given the `Competition Information`. + - Use cross-validation if possible, as it provides a more robust evaluation of the model's performance. 4. Submission File: - Save the final predictions as `submission.csv`, ensuring the format matches the competition requirements (refer to `sample_submission` in the Folder Description for the correct structure).