diff --git a/README.md b/README.md index a27c26e..cc01cca 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,31 @@ # 10-K Financial Analyzer -A web app that fetches the latest 10-K from SEC EDGAR for a given stock ticker, then uses **Item 7 (MD&A)** and **Item 8 (Financial Statements)** to produce a CFA-style analysis and key metrics. Powered by **Google Gemini**. +A web app that fetches the latest 10-K from SEC EDGAR for a given stock ticker and uses a **hybrid architecture**: **qualitative** analysis (Item 7 MD&A only) via **Google Gemini**, and **quantitative** metrics (Revenue, Net Income, Operating Cash Flow) from **yfinance**. CFA-style report and key financials in one place. --- ## Features -- **Selective section extraction:** Only Item 7 (MD&A) and Item 8 (Financial Statements) are sent to the API; PART I and Items 1–6 are pre-filtered to reduce tokens. -- **Smart chunking:** Long sections are trimmed to head + tail to stay within token limits while keeping high-signal content. -- **Two-step progress:** The UI shows Step 1 (download + extract) and Step 2 (Gemini analysis) so you can see where time is spent. -- **Analysis only mode:** Optional single API call (summary + CFA report only) to reduce rate-limit issues. -- **S&P 500 reference list:** A table of company names and tickers (sample) is shown at the bottom of the page for quick lookup. +- **Hybrid processing (qualitative + quantitative):** + - **Qualitative:** Only **Item 7 (MD&A)** is sent to Gemini for analysis of management’s strategy, market risks, and sentiment—no Item 8 (financial statements) to the AI, which cuts token use and avoids number hallucination. + - **Quantitative:** Financial metrics (Revenue, Net Income, Operating Cash Flow) are fetched directly from **yfinance**—fast, accurate, and no extra API tokens. +- **HTML cleansing:** Before sending Item 7 to the LLM, the app strips remaining HTML tags, collapses whitespace, and removes page numbers to compress tokens. +- **Selective extraction:** The 10-K is parsed with regex; only content from Item 7 onward is used for AI; PART I and Items 1–6 are dropped. +- **Smart chunking:** Long Item 7 text is trimmed to head + tail to stay within token limits. +- **Two-step progress:** Step 1 (download + extract Item 7), Step 2 (Gemini analysis + yfinance metrics). +- **Analysis only mode:** Optional hide for the metrics table (Gemini still runs once on Item 7). +- **S&P 500 reference list:** Sample table of company names and tickers at the bottom for quick lookup. -**Typical run time:** About **1–2 minutes** (roughly 1 minute with “Analysis only” enabled; up to 2 minutes with metrics). If the API is rate-limited, the app waits 60 seconds and retries automatically. +**Typical run time:** About **1–2 minutes** (one Gemini call; yfinance metrics are near-instant). If the API is rate-limited, the app waits 60 seconds and retries automatically. --- ## Tech Stack - **UI**: Streamlit -- **Data**: sec-edgar-downloader (SEC EDGAR) +- **Data**: sec-edgar-downloader (SEC EDGAR), **yfinance** (financial metrics) - **AI**: Google Gemini (google-generativeai) +- **Parsing / cleansing**: BeautifulSoup, regex --- @@ -29,13 +34,17 @@ A web app that fetches the latest 10-K from SEC EDGAR for a given stock ticker, During the initial development, I encountered a **429 Resource Exhausted** error due to the massive size of 10-K filings exceeding the LLM's token quota and rate limits. **Consultation & Architectural Pivot:** -After consulting with a senior software engineer, I re-architected the application to optimize token usage. Instead of processing the entire document, I implemented a **"Selective Section Extraction"** strategy. +After consulting with a senior software engineer, I re-architected the application to optimize token usage. The current design uses a **hybrid architecture** that separates qualitative and quantitative work. **Implemented Solution:** -- **Targeted Parsing:** Developed a regex-based parser to isolate only critical sections: Item 7 (MD&A) and Item 8 (Financial Statements). -- **Token Optimization:** Integrated a "Chunking & Filtering" logic to remove boilerplate legal text, sending only high-signal data to the Gemini API. -- **Efficiency:** This reduced token consumption by **over 80%**, ensuring stable performance within free-tier limits while maintaining analytical depth. +- **Selective section extraction:** A regex-based parser isolates Item 7 (MD&A) only for the AI; Item 8 is no longer sent to the LLM. +- **Hybrid processing:** + - **Qualitative (Gemini):** Item 7 only—strategy, risks, and sentiment. This drastically reduces tokens and avoids AI errors on exact figures. + - **Quantitative (yfinance):** Revenue, Net Income, and Operating Cash Flow are pulled from yfinance, so numbers are accurate and no tokens are spent on financial tables. +- **HTML cleansing:** Before sending Item 7 to Gemini, the app runs a cleansing step (BeautifulSoup + regex) to strip tags, collapse whitespace, and drop page numbers, further compressing tokens. +- **Chunking:** Long Item 7 text is trimmed to head + tail to stay within token limits. +- **Efficiency:** Token consumption is greatly reduced (one API call; no Item 8 in the prompt), and numeric accuracy is guaranteed via yfinance. For full technical notes and code references, see **[TECHNICAL_NOTES.md](./TECHNICAL_NOTES.md)**. @@ -78,11 +87,13 @@ Open the sidebar to set **Google API Key** and **SEC EDGAR Email**, then enter a ## Recent updates -- **Selective extraction & pre-filtering:** Regex-based extraction of Item 7 and Item 8 only; content before Item 7 is dropped to cut token use. -- **Smart chunking:** Sections over ~30k characters are reduced to head + tail before sending to Gemini. -- **Progress steps:** Step 1 (download + extract) and Step 2 (Gemini analysis, ~30–90s) with clear spinner messages. -- **S&P 500 list:** Bottom of the page now includes a sample table of S&P 500 companies with **company name** and **ticker** for easy reference. -- **Run time:** Results usually appear within 1–2 minutes under normal conditions. +- **Hybrid architecture:** Item 7 (MD&A) only is sent to Gemini for qualitative analysis (strategy, risks, sentiment). Financial metrics (Revenue, Net Income, Operating Cash Flow) come from **yfinance**—no Item 8 to the AI, fewer tokens, and accurate numbers. +- **HTML cleansing:** Pre-LLM step strips HTML remnants, extra whitespace, and page numbers to compress Item 7 text before sending to Gemini. +- **Selective extraction:** Regex-based extraction of Item 7 only for the API; content before Item 7 is dropped. +- **Smart chunking:** Item 7 over ~20k characters is reduced to head + tail before sending to Gemini. +- **Progress steps:** Step 1 (download + extract Item 7), Step 2 (Gemini qualitative analysis + yfinance metrics) with clear spinner messages. +- **S&P 500 list:** Sample table of S&P 500 companies (company name and ticker) at the bottom for quick reference. +- **Run time:** One Gemini call plus instant yfinance data; results typically within 1–2 minutes under normal conditions. --- diff --git a/app.py b/app.py index 30dec7b..c27d78d 100644 --- a/app.py +++ b/app.py @@ -1,8 +1,9 @@ """ -10-K Financial Analyzer (Google Gemini 1.5 Flash) -- Download 10-K from SEC EDGAR and extract text -- Analysis using Item 7 (MD&A) and Item 8 (Financial Statements) via Gemini 1.5 Flash (generous free tier, large context) -- CFA-style summary, key metrics table, and CFA Investment Report section +10-K Financial Analyzer (Google Gemini) — Hybrid Architecture +- Download 10-K from SEC EDGAR; extract Item 7 (MD&A) only for AI. +- Quantitative: financial metrics (Revenue, Net Income, Operating Cash Flow) from yfinance. +- Qualitative: Item 7 only to Gemini for strategic direction, risks, and sentiment analysis. +- HTML cleansing before sending text to LLM to minimise tokens. - All content in British English. """ @@ -18,20 +19,19 @@ import streamlit as st import pandas as pd from bs4 import BeautifulSoup -# Load .env if python-dotenv is available try: from dotenv import load_dotenv load_dotenv() except ImportError: pass + def get_edgar_downloader(): from sec_edgar_downloader import Downloader return Downloader def extract_text_from_html(html_path: Path) -> str: - """Extract plain text from an HTML file.""" try: with open(html_path, "r", encoding="utf-8", errors="replace") as f: soup = BeautifulSoup(f.read(), "lxml") @@ -44,7 +44,6 @@ def extract_text_from_html(html_path: Path) -> str: def extract_text_from_file(file_path: Path) -> str: - """Extract text by file extension (HTML or TXT).""" suf = file_path.suffix.lower() if suf in (".htm", ".html"): return extract_text_from_html(file_path) @@ -57,7 +56,6 @@ def extract_text_from_file(file_path: Path) -> str: return "" -# ---------- Selective Section Extraction (pre-filter: only Item 7 & 8, no PART I / ITEM 1–6) ---------- ITEM7_PATTERNS = [ r"Item\s+7\s*[.:]\s*Management['\u2019]s\s+Discussion\s+and\s+Analysis", r"ITEM\s+7\s*[.:]\s*Management['\u2019]s\s+Discussion", @@ -71,7 +69,6 @@ ITEM8_PATTERNS = [ def _find_section_start(text: str, patterns: list, item_num: int) -> int: - """Return start index of first matching pattern, or -1.""" for pat in patterns: m = re.search(pat, text, re.IGNORECASE) if m: @@ -81,13 +78,11 @@ def _find_section_start(text: str, patterns: list, item_num: int) -> int: def prefilter_after_item7(full_text: str) -> str: - """Drop PART I, ITEM 1–6; keep only from Item 7 onward to reduce noise and token use.""" start = _find_section_start(full_text, ITEM7_PATTERNS, 7) return full_text[start:] if start >= 0 else full_text def find_item_section(text: str, item_num: int, title_keywords: list) -> str: - """Extract only Item N section (regex-based). Used for Item 7 (MD&A) and Item 8 (Financial Statements).""" patterns = ITEM7_PATTERNS if item_num == 7 else ITEM8_PATTERNS start = _find_section_start(text, patterns, item_num) if start == -1: @@ -99,7 +94,6 @@ def find_item_section(text: str, item_num: int, title_keywords: list) -> str: if not match: return "" start = match.start() - # End at next "Item N" (next major section) next_item = re.search(r"\n\s*Item\s+\d+\s+", text[start + 100 :], re.IGNORECASE) if next_item: end = start + 100 + next_item.start() @@ -109,14 +103,10 @@ def find_item_section(text: str, item_num: int, title_keywords: list) -> str: def smart_chunk(section: str, max_chars: int = 30000, head_ratio: float = 0.5) -> str: - """ - If section exceeds max_chars, keep head and tail (quantitative data often at start/end). - Reduces tokens while preserving high-signal content. - """ if len(section) <= max_chars: return section head_size = int(max_chars * head_ratio) - tail_size = max_chars - head_size - 100 # reserve for separator + tail_size = max_chars - head_size - 100 return ( section[:head_size] + "\n\n[ ... middle omitted to stay within token limit ... ]\n\n" @@ -124,8 +114,40 @@ def smart_chunk(section: str, max_chars: int = 30000, head_ratio: float = 0.5) - ) +def clean_text_for_llm(text: str) -> str: + """ + Token-compression cleansing before sending to LLM: strip HTML remnants, + collapse whitespace, remove page numbers and excessive special characters. + """ + if not text or not text.strip(): + return "" + # Remove any remaining HTML tags (safe on plain text) + text = re.sub(r"<[^>]+>", " ", text) + # Collapse multiple spaces to one + text = re.sub(r"[ \t]+", " ", text) + # Normalise line endings and collapse many blank lines to at most two newlines + text = re.sub(r"\r\n?", "\n", text) + text = re.sub(r"\n{3,}", "\n\n", text) + lines = [] + for line in text.split("\n"): + line = line.strip() + # Drop lines that are only digits (page numbers) or only punctuation/dashes + if not line: + lines.append("") + continue + if re.fullmatch(r"\d+", line) or re.fullmatch(r"[\.\-\s\-]+", line): + continue + # Short boilerplate lines (e.g. "Page 1 of 2") — optional: drop very short lines that look like page refs + if re.match(r"^(page\s+\d+|\d+)\s*$", line, re.IGNORECASE) and len(line) < 20: + continue + lines.append(line) + # Rejoin and collapse again + result = "\n".join(lines) + result = re.sub(r"\n{3,}", "\n\n", result) + return result.strip() + + def find_downloaded_10k_path(download_root: Path, ticker: str) -> Optional[Path]: - """Return the path to the latest 10-K folder for the given ticker under download_root.""" ticker_upper = ticker.upper() for base in (download_root / "sec-edgar-filings", download_root): path_10k = base / ticker_upper / "10-K" @@ -148,7 +170,6 @@ def find_downloaded_10k_path(download_root: Path, ticker: str) -> Optional[Path] def get_main_10k_text(filing_dir: Path) -> str: - """Find the main document (HTML/TXT) in the 10-K folder and return its full text.""" all_text = [] for ext in ("*.htm", "*.html", "*.txt"): for path in filing_dir.rglob(ext): @@ -164,15 +185,12 @@ def get_main_10k_text(filing_dir: Path) -> str: return main_text -# ---------- Gemini 1.5 Flash: stable, generous free tier, good for large 10-K text ---------- GEMINI_MODEL = "gemini-2.0-flash" -# Wait 1 minute before retry when rate limited (free tier resets after a short period) RATE_LIMIT_WAIT_SEC = 60 DELAY_BETWEEN_CALLS_SEC = 8 def get_gemini_model(api_key: str): - """Return configured Gemini Flash model (generous free tier for large documents).""" import google.generativeai as genai genai.configure(api_key=api_key) return genai.GenerativeModel(GEMINI_MODEL) @@ -189,7 +207,6 @@ def _is_rate_limit_error(e: Exception) -> bool: def _generate_with_retry(model, content, generation_config, max_retries: int = 3): - """Call model.generate_content with retry on 429 (wait then retry up to max_retries times).""" last_err = None for attempt in range(max_retries + 1): try: @@ -203,62 +220,99 @@ def _generate_with_retry(model, content, generation_config, max_retries: int = 3 raise last_err -def get_ai_summary_and_report( - api_key: str, - full_text: str, - item7_text: str, - item8_text: str, - ticker: str, -) -> tuple[str, str]: +def get_metrics_from_yfinance(ticker: str) -> pd.DataFrame: """ - Produce detailed analysis and CFA report. Only Item 7 and Item 8 are sent (pre-filtered). - Sections are smart-chunked (head + tail) when long to keep token use low and avoid 429. + Quantitative data: fetch Revenue, Net Income, Operating Cash Flow from yfinance + (no LLM; fast and accurate). Returns a DataFrame suitable for Streamlit display. + """ + try: + import yfinance as yf + except ImportError: + return pd.DataFrame() + try: + t = yf.Ticker(ticker.upper()) + financials = t.financials # annual income statement + cashflow = t.cashflow # annual cash flow + if financials is None or financials.empty: + return pd.DataFrame() + # Prefer common index names (yfinance varies by region) + rev_row = None + for name in ("Total Revenue", "Revenue", "Net Revenue", "Operating Revenue"): + if name in financials.index: + rev_row = financials.loc[name] + break + ni_row = None + for name in ("Net Income", "Net Income Common Stockholders", "Net Income Including Noncontrolling Interests"): + if name in financials.index: + ni_row = financials.loc[name] + break + ocf_row = None + if cashflow is not None and not cashflow.empty: + for name in ("Operating Cash Flow", "Cash From Operating Activities", "Cash From Operations"): + if name in cashflow.index: + ocf_row = cashflow.loc[name] + break + # Align by date (columns are often datetime) + dates = financials.columns.tolist() + if not dates: + return pd.DataFrame() + # Sort descending (most recent first) and take up to 5 years + dates = sorted(dates, reverse=True)[:5] + cashflow_cols = list(cashflow.columns) if cashflow is not None and not cashflow.empty else [] + data = {} + for d in dates: + yr = d.year if hasattr(d, "year") else int(str(d)[:4]) + rev_val = (rev_row[d] / 1e6) if rev_row is not None and d in rev_row.index else None + ni_val = (ni_row[d] / 1e6) if ni_row is not None and d in ni_row.index else None + ocf_val = None + if ocf_row is not None: + if d in ocf_row.index: + ocf_val = ocf_row[d] / 1e6 + else: + for c in cashflow_cols: + cy = c.year if hasattr(c, "year") else int(str(c)[:4]) + if cy == yr: + ocf_val = ocf_row[c] / 1e6 + break + data[yr] = {"Revenue": rev_val, "Net Income": ni_val, "Operating Cash Flow": ocf_val} + df = pd.DataFrame(data).T + df.index.name = "Fiscal Year" + df = df.astype(float).round(2) + return df + except Exception: + return pd.DataFrame() + + +def get_ai_summary_and_report(api_key: str, item7_text: str, ticker: str) -> tuple[str, str]: + """ + Qualitative only: send Item 7 (MD&A) to Gemini. Focus on strategic direction, + market risks, and sentiment—not on summarising financial statement numbers. """ model = get_gemini_model(api_key) + item7_text = clean_text_for_llm(item7_text) + item7_text = smart_chunk(item7_text, max_chars=20000) - # Smart chunking: when over limit, keep head + tail (figures often at start/end) - max_chars_per_section = 30000 - item7_text = smart_chunk(item7_text, max_chars=max_chars_per_section) - item8_text = smart_chunk(item8_text, max_chars=max_chars_per_section) + user_prompt = f"""You are a CFA charterholder and senior equity analyst. Use British English. - user_prompt = f"""You are a CFA charterholder and senior equity analyst. Use British English. Omit unnecessary qualifiers and filler; focus on figures, risks, and material facts. +The text below is Item 7 (Management's Discussion and Analysis) only from the 10-K for company ticker: {ticker}. Do NOT ask for financial statements or numbers—this is a qualitative analysis. -Analyse the following 10-K excerpts for company ticker: {ticker}. The text below contains ONLY Item 7 (MD&A) and Item 8 (Financial Statements)—other sections have been pre-filtered out. +Your task: +1. **Strategic direction**: How does management describe its strategy, priorities, and capital allocation? What are the main growth drivers or initiatives? +2. **Market and business risks**: What material risks (competitive, regulatory, operational, macro) does management emphasise? Be specific and cite the wording where relevant. +3. **Tone (Sentiment)**: Overall, is the tone of MD&A more positive, cautious, or negative? Highlight 2–3 phrases or themes that support your view. -Use the provided text to produce a thorough, evidence-based analysis. Cite specific numbers and risk disclosures where relevant. +Then write a "CFA INVESTMENT REPORT" section with: +- **Executive Summary**: 2–3 sentences on the company's narrative and management's message. +- **Investment Thesis**: Key strengths and catalysts from the discussion. +- **Key Risks to the Thesis**: Main downside risks from the text. +- **Conclusion**: Balanced wrap-up. -First, write a "DETAILED ANALYSIS" section with exactly three paragraphs (use subheadings): -1. **Financial Health**: Liquidity (current ratio, cash position, credit facilities), leverage (debt/equity, interest coverage), capital structure, and any covenant or refinancing risks. Cite figures from the statements. -2. **Profitability**: Revenue and earnings trends, margins (gross, operating, net), earnings quality (e.g. non-GAAP adjustments, one-time items), and sustainability of earnings. Use numbers from the 10-K. -3. **Key Risks**: Material risk factors from MD&A and notes (market, credit, operational, legal, ESG if material). Be specific; quote or paraphrase the filing. +Keep the entire response in British English. Use clear section headers. Do not invent figures—only refer to what is in the text.""" -Then, write a "CFA INVESTMENT REPORT" section in the style of a formal sell-side or buy-side investment memo. Include these subsections with clear headings: -- **Executive Summary**: 2–3 sentences on the company's position and your high-level view. -- **Investment Thesis**: Why an investor might consider this company (strengths, catalysts). Be specific. -- **Valuation Considerations**: What to watch (multiples, growth, margins, capital allocation). No exact price target required. -- **Key Risks to the Thesis**: Main downside risks that could invalidate the thesis. -- **Conclusion**: One short paragraph with a balanced wrap-up (e.g. Hold/Overweight/Underweight context and what would change your view). - -Keep the entire response in British English. Use clear section headers (e.g. ## or **) and professional language.""" - - full_content = f"""--- Item 7. Management's Discussion and Analysis (full or extended excerpt) --- - -{item7_text} - ---- Item 8. Financial Statements and Notes (full or extended excerpt) --- - -{item8_text} - ---- - -{user_prompt}""" + full_content = f"""--- Item 7. Management's Discussion and Analysis (MD&A) ---\n\n{item7_text}\n\n---\n\n{user_prompt}""" try: - response = _generate_with_retry( - model, - full_content, - {"temperature": 0.3, "max_output_tokens": 8192}, - ) + response = _generate_with_retry(model, full_content, {"temperature": 0.3, "max_output_tokens": 8192}) except Exception as api_err: if _is_rate_limit_error(api_err): raise RuntimeError("Rate limit exceeded. Please try again in a few minutes.") from api_err @@ -268,69 +322,18 @@ Keep the entire response in British English. Use clear section headers (e.g. ## return "No analysis generated.", "No report generated." text = response.text.strip() - - # Split into "DETAILED ANALYSIS" and "CFA INVESTMENT REPORT" if the model used those headers - detailed = "" - report = "" - if "CFA INVESTMENT REPORT" in text.upper() or "CFA Investment Report" in text: + detailed, report = text, "" + if "CFA INVESTMENT REPORT" in text.upper(): parts = re.split(r"\n\s*(?:CFA INVESTMENT REPORT|CFA Investment Report)\s*\n", text, maxsplit=1, flags=re.IGNORECASE) detailed = (parts[0].replace("DETAILED ANALYSIS", "").strip() if parts else "").strip() or text report = parts[1].strip() if len(parts) > 1 else "" - if not detailed: - detailed = text else: - detailed = text report = "(CFA Investment Report section not clearly separated; full analysis above.)" return detailed, report -def get_metrics_table_from_ai(api_key: str, item8_text: str, ticker: str) -> pd.DataFrame: - """Extract Revenue, Net Income, Operating Cash Flow from Item 8 only (pre-filtered).""" - model = get_gemini_model(api_key) - excerpt = smart_chunk(item8_text, max_chars=25000) - - prompt = f"""You are a financial analyst. From the 10-K Item 8 excerpt below for company {ticker}, extract the following for the most recent 3–5 fiscal years. Focus only on figures; omit filler text. -- Revenue (or Net sales) -- Net Income (or Net earnings attributable to common shareholders) -- Cash flows from operating activities (Operating Cash Flow) - -Reply with ONLY a single JSON object, no other text. Use fiscal years as keys (e.g. "2023", "2022", "2021"). -Format: -{{"Revenue": {{"2023": 123.45, "2022": 100.0}}, "Net Income": {{"2023": 20.0, "2022": 18.0}}, "Operating Cash Flow": {{"2023": 25.0, "2022": 22.0}}}} -Use numbers in millions (e.g. 394328 for $394,328 million). If a value is not found, use null. - -Item 8 excerpt: - -{excerpt}""" - - try: - response = _generate_with_retry( - model, - prompt, - {"temperature": 0.1, "max_output_tokens": 1024}, - ) - except Exception as api_err: - if _is_rate_limit_error(api_err): - raise RuntimeError("Rate limit exceeded. Please try again in a few minutes.") from api_err - raise - - if not response or not response.text: - return pd.DataFrame() - - text = response.text.strip() - json_match = re.search(r"\{[\s\S]*\}", text) - if not json_match: - return pd.DataFrame() - try: - data = json.loads(json_match.group()) - return pd.DataFrame(data) - except Exception: - return pd.DataFrame() - - def download_and_extract_sections(ticker: str, email: str) -> tuple[str, str, str]: - """Download 10-K, pre-filter, extract Item 7 & 8 only. Returns (full_text, item7, item8).""" Downloader = get_edgar_downloader() with tempfile.TemporaryDirectory() as tmpdir: download_root = Path(tmpdir) @@ -342,57 +345,37 @@ def download_and_extract_sections(ticker: str, email: str) -> tuple[str, str, st full_text = get_main_10k_text(filing_dir) if not full_text: raise ValueError("Could not extract text from the 10-K.") - text_from_item7 = prefilter_after_item7(full_text) item7 = find_item_section(text_from_item7, 7, ["Management's Discussion", "MD&A", "Analysis"]) item8 = find_item_section(text_from_item7, 8, ["Financial Statements", "Consolidated"]) - if not item7: - item7 = smart_chunk(text_from_item7[:120000], max_chars=30000) + item7 = smart_chunk(text_from_item7[:120000], max_chars=20000) if not item8: remainder = text_from_item7[100000:220000] if len(text_from_item7) > 100000 else text_from_item7 - item8 = smart_chunk(remainder, max_chars=30000) - + item8 = smart_chunk(remainder, max_chars=20000) return full_text, item7, item8 def run_analysis(ticker: str, api_key: str, email: str, analysis_only: bool = False) -> tuple[str, str, str, pd.DataFrame]: - """Download 10-K, extract Item 7/8, call Gemini; return summary, report, full_text, metrics table.""" - full_text, item7, item8 = download_and_extract_sections(ticker, email) - - detailed_summary, cfa_report = get_ai_summary_and_report(api_key, full_text, item7, item8, ticker) + full_text, item7, _ = download_and_extract_sections(ticker, email) + detailed_summary, cfa_report = get_ai_summary_and_report(api_key, item7, ticker) if analysis_only: df_metrics = pd.DataFrame() else: - time.sleep(DELAY_BETWEEN_CALLS_SEC) - df_metrics = get_metrics_table_from_ai(api_key, item8, ticker) - + df_metrics = get_metrics_from_yfinance(ticker) return detailed_summary, cfa_report, full_text, df_metrics # ---------- Streamlit UI ---------- st.set_page_config(page_title="10-K Financial Analyzer", layout="wide") st.title("10-K Financial Analyzer") -st.caption("Download 10-K from SEC EDGAR; view detailed analysis and a CFA-style investment report. Powered by Google Gemini.") +st.caption("Hybrid: 10-K Item 7 (MD&A) → Gemini for sentiment & risks; financial metrics from yfinance. British English.") with st.sidebar: st.header("Settings") - google_api_key = st.text_input( - "Google API Key (Gemini)", - type="password", - value=os.environ.get("GOOGLE_API_KEY", ""), - help="Obtain from https://aistudio.google.com/apikey (Google AI Studio).", - ) - email = st.text_input( - "SEC EDGAR Email Address", - value=os.environ.get("SEC_EDGAR_EMAIL", ""), - help="Required for SEC programmatic download policy compliance.", - ) - analysis_only = st.checkbox( - "Analysis only (1 API call)", - value=False, - help="Skip metrics table to use only 1 API call. Turn on if you often hit rate limits.", - ) + google_api_key = st.text_input("Google API Key (Gemini)", type="password", value=os.environ.get("GOOGLE_API_KEY", ""), help="Obtain from https://aistudio.google.com/apikey") + email = st.text_input("SEC EDGAR Email Address", value=os.environ.get("SEC_EDGAR_EMAIL", ""), help="Required for SEC programmatic download policy compliance.") + analysis_only = st.checkbox("Analysis only (1 API call)", value=False, help="Skip metrics table to use only 1 API call.") st.session_state["google_api_key"] = google_api_key st.session_state["email"] = email st.session_state["analysis_only"] = analysis_only @@ -401,7 +384,6 @@ ticker = st.text_input("Stock Ticker (e.g. AAPL, MSFT)", value="AAPL", max_chars if not ticker: st.info("Enter a ticker and click 'Run Analysis', or pick one from the S&P 500 list below.") -# S&P 500 sample: (Company name, Ticker) – shown at bottom SP500_SAMPLE = [ ("Apple Inc.", "AAPL"), ("Microsoft Corporation", "MSFT"), ("Amazon.com Inc.", "AMZN"), ("NVIDIA Corporation", "NVDA"), ("Alphabet Inc. (Google)", "GOOGL"), ("Meta Platforms Inc. (Facebook)", "META"), @@ -409,30 +391,13 @@ SP500_SAMPLE = [ ("Visa Inc.", "V"), ("UnitedHealth Group Inc.", "UNH"), ("Procter & Gamble Co.", "PG"), ("Exxon Mobil Corporation", "XOM"), ("Johnson & Johnson", "JNJ"), ("Mastercard Inc.", "MA"), ("Chevron Corporation", "CVX"), ("Home Depot Inc.", "HD"), ("Merck & Co. Inc.", "MRK"), - ("AbbVie Inc.", "ABBV"), ("Costco Wholesale Corporation", "COST"), - ("PepsiCo Inc.", "PEP"), ("Coca-Cola Company", "KO"), ("Pfizer Inc.", "PFE"), - ("Walmart Inc.", "WMT"), ("Netflix Inc.", "NFLX"), ("Adobe Inc.", "ADBE"), - ("Salesforce Inc.", "CRM"), ("Comcast Corporation", "CMCSA"), ("Cisco Systems Inc.", "CSCO"), + ("AbbVie Inc.", "ABBV"), ("Costco Wholesale Corporation", "COST"), ("PepsiCo Inc.", "PEP"), + ("Coca-Cola Company", "KO"), ("Pfizer Inc.", "PFE"), ("Walmart Inc.", "WMT"), ("Netflix Inc.", "NFLX"), + ("Adobe Inc.", "ADBE"), ("Salesforce Inc.", "CRM"), ("Comcast Corporation", "CMCSA"), ("Cisco Systems Inc.", "CSCO"), ("Oracle Corporation", "ORCL"), ("Intel Corporation", "INTC"), ("American Express Company", "AXP"), ("Bank of America Corp.", "BAC"), ("Wells Fargo & Company", "WFC"), ("Verizon Communications Inc.", "VZ"), - ("AT&T Inc.", "T"), ("Disney (Walt Disney Co.)", "DIS"), ("Nike Inc.", "NKE"), - ("McDonald's Corporation", "MCD"), ("Starbucks Corporation", "SBUX"), ("Goldman Sachs Group Inc.", "GS"), - ("Morgan Stanley", "MS"), ("Boeing Company", "BA"), ("Caterpillar Inc.", "CAT"), - ("3M Company", "MMM"), ("Honeywell International Inc.", "HON"), ("IBM (International Business Machines)", "IBM"), - ("Qualcomm Inc.", "QCOM"), ("Texas Instruments Inc.", "TXN"), ("Amgen Inc.", "AMGN"), - ("Gilead Sciences Inc.", "GILD"), ("Bristol-Myers Squibb Company", "BMY"), ("Eli Lilly and Company", "LLY"), - ("Union Pacific Corporation", "UNP"), ("Lockheed Martin Corporation", "LMT"), ("Raytheon Technologies Corp.", "RTX"), - ("Target Corporation", "TGT"), ("Lowe's Companies Inc.", "LOW"), ("Booking Holdings Inc.", "BKNG"), - ("PayPal Holdings Inc.", "PYPL"), ("Broadcom Inc.", "AVGO"), ("Schlumberger Ltd.", "SLB"), - ("ConocoPhillips", "COP"), ("Phillips 66", "PSX"), - ("Ford Motor Company", "F"), ("General Motors Company", "GM"), ("General Electric Company", "GE"), - ("FedEx Corporation", "FDX"), ("United Parcel Service Inc.", "UPS"), ("Delta Air Lines Inc.", "DAL"), - ("American Airlines Group Inc.", "AAL"), ("Southwest Airlines Co.", "LUV"), - ("Abbott Laboratories", "ABT"), ("Thermo Fisher Scientific Inc.", "TMO"), ("Danaher Corporation", "DHR"), - ("Accenture plc", "ACN"), ("Intuit Inc.", "INTU"), - ("ServiceNow Inc.", "NOW"), ("Workday Inc.", "WDAY"), ("Snowflake Inc.", "SNOW"), - ("Zoom Video Communications Inc.", "ZM"), ("Spotify Technology S.A.", "SPOT"), ("Uber Technologies Inc.", "UBER"), - ("Airbnb Inc.", "ABNB"), ("Moderna Inc.", "MRNA"), ("Regeneron Pharmaceuticals Inc.", "REGN"), + ("AT&T Inc.", "T"), ("Disney (Walt Disney Co.)", "DIS"), ("Nike Inc.", "NKE"), ("McDonald's Corporation", "MCD"), + ("Starbucks Corporation", "SBUX"), ("Goldman Sachs Group Inc.", "GS"), ("Morgan Stanley", "MS"), ] st.caption("Select a ticker above or choose from the list below.") @@ -444,37 +409,35 @@ if st.button("Run Analysis"): api_key = st.session_state.get("google_api_key", "") email = st.session_state.get("email", "") if not api_key: - st.error("Please enter your Google API Key (Gemini) in Settings. You may also set GOOGLE_API_KEY in a .env file.") + st.error("Please enter your Google API Key (Gemini) in Settings.") st.stop() if not email: st.error("Please enter your SEC EDGAR email address in Settings.") st.stop() - analysis_only = st.session_state.get("analysis_only", False) try: - with st.spinner("Step 1/2: Downloading 10-K and extracting Item 7 & 8 (selective sections only)..."): - full_text, item7, item8 = download_and_extract_sections(ticker, email) - - with st.spinner("Step 2/2: Running Gemini analysis (typically 30–90s; if rate limited, we wait 60s then retry)..."): - detailed_summary, cfa_report = get_ai_summary_and_report(api_key, full_text, item7, item8, ticker) + with st.spinner("Step 1/2: Downloading 10-K and extracting Item 7 (MD&A)..."): + full_text, item7, _ = download_and_extract_sections(ticker, email) + with st.spinner("Step 2/2: Running Gemini (qualitative analysis) and fetching financial metrics..."): + detailed_summary, cfa_report = get_ai_summary_and_report(api_key, item7, ticker) if analysis_only: df_metrics = pd.DataFrame() else: - time.sleep(DELAY_BETWEEN_CALLS_SEC) - df_metrics = get_metrics_table_from_ai(api_key, item8, ticker) + df_metrics = get_metrics_from_yfinance(ticker) st.success("Analysis complete.") - st.subheader("Detailed Analysis (Financial Health, Profitability, Key Risks)") + st.subheader("Detailed Analysis (Strategy, Risks, Sentiment — from Item 7 MD&A)") st.markdown(detailed_summary) st.subheader("CFA Investment Report") st.markdown(cfa_report) - st.subheader("Key Financial Metrics (Revenue, Net Income, Operating Cash Flow)") + st.subheader("Key Financial Metrics (Revenue, Net Income, Operating Cash Flow) — from yfinance") if not df_metrics.empty: st.dataframe(df_metrics, use_container_width=True) + st.caption("Values in millions (USD). Source: yfinance.") elif analysis_only: - st.info("Metrics skipped (Analysis only mode). Turn off 'Analysis only' in Settings to fetch metrics.") + st.info("Metrics skipped (Analysis only mode).") else: - st.info("No metrics extracted. Check the full Item 8 text.") + st.info("No metrics available for this ticker from yfinance.") with st.expander("View excerpt of extracted 10-K text"): st.text(full_text[:15000] + ("..." if len(full_text) > 15000 else "")) @@ -485,29 +448,28 @@ if st.button("Run Analysis"): except RuntimeError as e: st.error(str(e)) if analysis_only: - st.warning("You already have **Analysis only** on (1 API call). The limit is on Google's side — wait **2–5 minutes** without clicking, then press Run Analysis again.") + st.warning("You already have Analysis only on. Wait 2–5 minutes, then try again.") else: - st.info("Wait 2–5 minutes, then try again. Or enable 'Analysis only (1 API call)' in Settings to reduce usage.") + st.info("Wait 2–5 minutes, or enable Analysis only (1 API call) in Settings.") except Exception as e: err_msg = str(e).lower() if "429" in err_msg or ("resource" in err_msg and "exhausted" in err_msg): st.error("Rate limit exceeded. Please try again in a few minutes.") - if analysis_only: - st.warning("You already have **Analysis only** on. Google's free tier limit is reached — wait **2–5 minutes**, then press Run Analysis again (no need to change settings).") - else: - st.info("Wait 2–5 minutes, then retry. Or enable **Analysis only (1 API call)** in the sidebar.") + st.info("Wait 2–5 minutes, or enable **Analysis only (1 API call)** in the sidebar.") elif "404" in err_msg or "not found" in err_msg: - st.error("The selected model is not available. Please try again later or check Google AI Studio for available models.") + st.error("The selected model is not available. Check Google AI Studio for available models.") + elif "timeout" in err_msg or "retryerror" in err_msg or "600" in err_msg: + st.error("Request timed out. The API took too long to respond.") + st.info("Try again, or enable **Analysis only (1 API call)** to send less data.") else: st.error("An error occurred. Please try again later.") st.caption("If the problem persists, check your API key and internet connection.") with st.expander("Error details (for troubleshooting)"): st.code(repr(e), language="text") - st.caption("Share this with support if the issue continues.") st.divider() st.subheader("S&P 500 companies (sample) — Company name & Ticker") -st.caption("Click a row to copy the ticker, or type it in the box above.") +st.caption("Type a ticker from the list into the box above.") df_sp = pd.DataFrame(SP500_SAMPLE, columns=["Company name", "Ticker"]) with st.expander("Show list", expanded=True): st.dataframe(df_sp, use_container_width=True, hide_index=True) diff --git a/find_toc.py b/find_toc.py new file mode 100644 index 0000000..4ac3162 --- /dev/null +++ b/find_toc.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +""" +Find Table of Contents in an SEC EDGAR 10-K HTML document. +Usage: + python find_toc.py + python find_toc.py "https://www.sec.gov/Archives/edgar/data/2012383/000095017025026584/blk-20241231.htm" +""" + +import os +import re +import sys +from urllib.request import Request, urlopen +from bs4 import BeautifulSoup + +import warnings +from bs4 import XMLParsedAsHTMLWarning +warnings.filterwarnings("ignore", category=XMLParsedAsHTMLWarning) + +try: + from dotenv import load_dotenv + load_dotenv() +except ImportError: + pass + +# SEC requires a descriptive User-Agent (use SEC_EDGAR_EMAIL from .env if set) +_ua = os.environ.get("SEC_EDGAR_EMAIL", "CompanyName contact@example.com") +HEADERS = { + "User-Agent": _ua, + "Accept": "text/html,application/xhtml+xml", +} + +DEFAULT_URL = "https://www.sec.gov/Archives/edgar/data/2012383/000095017025026584/blk-20241231.htm" + + +def fetch_html(url: str) -> str: + req = Request(url, headers=HEADERS) + with urlopen(req, timeout=30) as resp: + return resp.read().decode("utf-8", errors="replace") + + +def find_toc(html: str) -> list[tuple[str, str]]: + """ + Find Table of Contents entries. Returns list of (label, href or ""). + """ + soup = BeautifulSoup(html, "html.parser") + toc_entries = [] + + # 1) Look for element with id/class containing toc, contents, table + for attr in ("id", "class"): + for tag in soup.find_all(True, **{attr: re.compile(r"toc|content|table\s*of\s*content", re.I)}): + if not tag.get("class") or (attr == "class" and not any(re.search(r"toc|content", c, re.I) for c in tag.get("class", []))): + if attr == "id" and not re.search(r"toc|content", tag.get("id", ""), re.I): + continue + # Collect links inside this block (Item 1, Item 7, etc.) + for a in tag.find_all("a", href=True): + text = a.get_text(strip=True) or "" + href = a["href"].strip() + if text and (re.search(r"item\s*\d|part\s*[IV]+", text, re.I) or href.startswith("#")): + toc_entries.append((text, href)) + if toc_entries: + return toc_entries + + # 2) Look for heading "Table of Contents" or "Contents" and take following list/links + for tag in soup.find_all(string=re.compile(r"table\s*of\s*contents|^contents\s*$", re.I)): + parent = tag.parent + if parent is None: + continue + # Next sibling or parent's next sibling + block = parent.find_next_sibling() or parent.parent.find_next_sibling() if parent.parent else None + if block: + for a in block.find_all("a", href=True): + text = a.get_text(strip=True) or "" + href = a["href"].strip() + if text: + toc_entries.append((text, href)) + if toc_entries: + return toc_entries + + # 3) Collect all links that look like TOC (anchor to item/part) + for a in soup.find_all("a", href=True): + href = a["href"].strip() + text = a.get_text(strip=True) or "" + if not text: + continue + # Anchor link like #item1, #part1, or text like "Item 1", "Part I" + if href.startswith("#") and re.search(r"item\s*\d|part\s*[IV]+|\d+\.", text, re.I): + toc_entries.append((text, href)) + if re.search(r"#item\s*\d|#part", href, re.I): + toc_entries.append((text, href)) + + # 4) Fallback: find any list or div that has multiple "Item N" links + for container in soup.find_all(["div", "nav", "section", "ul", "ol"]): + links = container.find_all("a", href=True) + if len(links) < 3: + continue + items = [] + for a in links: + t = a.get_text(strip=True) + h = a["href"].strip() + if re.search(r"item\s*\d|part\s*[IV]|\d+\.", t, re.I) or (h.startswith("#") and t): + items.append((t, h)) + if len(items) >= 3: + return items + + return toc_entries + + +def main(): + url = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_URL + print(f"Fetching: {url}\n") + + try: + html = fetch_html(url) + except Exception as e: + print(f"Error fetching URL: {e}") + sys.exit(1) + + toc = find_toc(html) + if not toc: + print("No Table of Contents block found. Showing links that look like Item/Part anchors:\n") + soup = BeautifulSoup(html, "html.parser") + for a in soup.find_all("a", href=True): + t = a.get_text(strip=True) + h = a["href"] + if h.startswith("#") and (t or re.search(r"item|part", h, re.I)): + toc.append((t or h, h)) + toc = toc[:80] # limit + + print("Table of Contents (or relevant links):") + print("-" * 60) + for label, href in toc: + print(f" {label or '(no text)'}\t{href}") + print("-" * 60) + print(f"Total: {len(toc)} entries.") + + +if __name__ == "__main__": + main() diff --git a/requirements.txt b/requirements.txt index 99425e5..8f0ab43 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,3 +6,4 @@ requests>=2.31.0 pandas>=2.0.0 lxml>=4.9.0 python-dotenv>=1.0.0 +yfinance>=0.2.40