mirror of
https://github.com/xavierchuan/FX-ML-Trading-Engine.git
synced 2026-08-09 15:40:55 +00:00
52 lines
11 KiB
Plaintext
52 lines
11 KiB
Plaintext
|
ó
|
|||
|
|
žsi— ã óø • S r SSKJr SSKrSSKrSSKJrJr SSKJr SSK J
|
|||
|
|
r
|
|||
|
|
J r J r SSK
|
|||
|
|
rSSKrSSKJr / SQrS r\" S
|
|||
|
|
5 rSS jrSS jrSS
|
|||
|
|
jrSS jrSS jrSS jrSS jr SS jr \ S:X a \ " 5 gg) zŸ
|
|||
|
|
Validate dataset quality (missing data, duplicate timestamps, gaps, outliers).
|
|||
|
|
|
|||
|
|
Example:
|
|||
|
|
python scripts/validate_dataset.py --path data/raw/EURUSD_H1.csv
|
|||
|
|
é )Ú annotationsN)ÚdatetimeÚtimezone)ÚPath)ÚDictÚOptionalÚTuple)Úlogger)ÚtsÚtimeÚ timestampr Údatezdata/_manifest.jsonzresults/data_qualityc óä • [ R " SS9n U R SSSS9 U R S[ SS 9 U R S
|
|||
|
|
S S S 9 U R S [ S
|
|||
|
|
SS9 U R 5 $ )NzValidate dataset quality.)Ú descriptionz--pathTz)Dataset file path (relative or absolute).)ÚrequiredÚhelpz
|
|||
|
|
--manifestz(Optional manifest JSON to enrich report.)Údefaultr z--outputz'Optional explicit path for JSON report.z--max-outlier-zg @z+Z-score threshold to flag numeric outliers.)Útyper r )ÚargparseÚArgumentParserÚ add_argumentÚDEFAULT_MANIFESTÚfloatÚ
|
|||
|
|
parse_args)Úparsers ÚU/Users/chuan/Documents/Projects/FX_Backtest/QuantResearch/scripts/validate_dataset.pyr r s• € Ü
|
|||
|
|
×
|
|||
|
|
$Ò
|
|||
|
|
$Ð1LÑ
|
|||
|
|
M€FØ
|
|||
|
|
×ј¨4Ð6aÐÑbØ
|
|||
|
|
×ÑØÜ Ø
|
|||
|
|
7ð ñ ð
|
|||
|
|
×ÑØØØ
|
|||
|
|
6ð ñ ð
|
|||
|
|
×ÑØÜ
|
|||
|
|
ØØ
|
|||
|
|
:ð ñ ð × Ñ Ó Ð ó c óÒ • [ H nXR ; d M Us $ U R H7 n[ R R R X 5 ( d M5 Us $ g )N)Ú TIME_COLUMNSÚcolumnsÚpdÚapiÚtypesÚis_datetime64_any_dtype)ÚdfÚcols r Údetect_time_columnr' 1 sP € ßˆØ —*‘*Õ ØŠJñ ð �zŒzˆÜ
|
|||
|
|
�6‰6�<‰<× /Ñ /°±× 8Ó 8ØŠJñ ð r c óx • [ R " U SSS9R 5 nUR S: a g [ R
|
|||
|
|
" UR R S5 R [ R 5 S- 5 nX"S:„ nUR S:X a g [ [ R " U5 5 $ )NTÚcoerce©ÚutcÚerrorsé údatetime64[ns]é Êš;r ) r! Ú to_datetimeÚdropnaÚsizeÚnpÚdiffÚvaluesÚastypeÚint64r Úmedian)Úseriesr Údiffss r Úinfer_interval_secondsr; ; s‹ € Ü � Š ˜ D°Ñ :× AÑ AÓ C€BØ ‡w�w�ƒ{ØÜ �GŠG�B—I‘I×$Ñ$Ð%5Ó6×=Ñ=¼b¿h¹hÓGÈ=ÑXÓ Y€EØ ˜!‘)Ñ €EØ ‡z�z�QƒØÜ ”—’˜5Ó!Ó "Ð"r c ó
|
|||
|
|
• U R 5 ( d g U R SSS9 n[ R " U5 nS S S 5 [ U5 n[ UR 5 R [ R " 5 5 5 nWR S/ 5 H nUR S5 XV4; d M Us $ g ! , ( d f Nˆ= f! [ a& n[
|
|||
|
|
R " SU SU 35 S nAg S nAff = f)NÚrúutf-8©ÚencodingzFailed to parse manifest z: ÚfilesÚpath)
|
|||
|
|
ÚexistsÚopenÚjsonÚloadÚ Exceptionr
|
|||
|
|
ÚwarningÚstrÚresolveÚ relative_tor ÚcwdÚget)Ú
|
|||
|
|
manifest_pathÚ dataset_pathÚfhÚmanifestÚexcÚrelÚrel_altÚentrys r Úload_manifest_entryrV F s× € Ø × Ñ × !Ñ !ØðØ
|
|||
|
|
×
|
|||
|
|
Ñ
|
|||
|
|
¨gÐ
|
|||
|
|
Ñ
|
|||
|
|
6¸"Ü—y’y “}ˆH÷ 7ô
|
|||
|
|
ˆlÓ
|
|||
|
|
€CÜ�,×&Ñ&Ó(×4Ñ4´T·X²X³ZÓ@ÓA€GØ— ‘ ˜g rÖ*ˆØ �9‰9�VÓ Õ .ØŠLñ +ð ÷ 7Õ
|
|||
|
|
6ûä ó Ü�ŠÐ2°=°/ÀÀCÀ5ÐIÔJÜûðús3 ˜C ¨C¿C Ã
|
|||
|
|
CÃ C ÃC Ã
|
|||
|
|
DÃ C=Ã=Dc óª • 0 nU R [ R /S9R nUR ( a U$ U H� nX R 5 nUR ( a M( [ R " XUR 5 -
|
|||
|
|
UR SS9=( d S- 5 n[ Xa:„ R 5 5 nU( d M‹ XrU' M‘ U$ )N)Úincludeé )Úddof) Ú
|
|||
|
|
select_dtypesr3 Únumberr Úemptyr1 ÚabsÚmeanÚstdÚintÚsum)r% Ú thresholdÚoutliersÚ numeric_colsr& r9 ÚzscoresÚcounts r Úsummarize_numeric_outliersrh W s« € Ø !€HØ×#Ñ#¬R¯Y©Y¨KÐ#Ð8×@Ñ@€LØ××؈ۈؑ— ‘ Ó!ˆØ �<�<Ù Ü—&’&˜&§;¡;£=Ñ0°V·Z±ZÀQ°ZÐ5G×5LÈ1ÑMÓNˆÜ�WÑ(×-Ñ-Ó/Ó0ˆß ˆ5Ø !�S‹Mñ ð €Or c óB ^• [ R " U 5 n[ UR S 5 mUR 5 R 5 R
|
|||
|
|
5 n[ U5 nSnSnS nU( aô [ R " X5 SSS9n [ U R 5 R 5 5 nU R 5 R 5 n
|
|||
|
|
[ U
|
|||
|
|
5 nU( a… [ R " U
|
|||
|
|
R R! S5 R! [ R" 5 S- 5 n Un X»U S- :„ n
|
|||
|
|
[% [' U
|