51 Indicators × 5000 Stocks: Engineering Practices for Full-Market Technical Scanning

Every trading day, the A-share market has 5,000+ stocks that need to be reviewed, and each stock requires dozens of technical indicators to be computed. Manual screening is completely inadequate at this scale. Programmatic scanning is not a question of "whether it can be done," but an engineering problem of "how to organize the data pipeline, indicator computation, signal scanning, and result presentation clearly."

This article is based on the real source code of stock-analyzer, and walks through its overall architecture and key trade-offs — from data ingestion to AI-assisted stock picking.

💡 The two things most worth seeing in this toolkit:

1. Ask the full 5,000-stock market in natural language — the system translates your question into SQL and runs it read-only against real quote data, returning results and charts (see Section 7);
2. The 51-indicator × 5,000-stock full-market scan is sustained by incremental ETL + three-library separation, not by swapping the compute kernel.

---

1. System Positioning: An Analytics Toolkit, Not a Quant Platform

The boundary of stock-analyzer is clear: it is an open-source analytics toolkit for A-share daily data, not a live-trading quantitative platform. It bundles several relatively independent capability lines:

• ETL data pipeline: ingest daily K-lines from a free data source and compute 51 technical indicators;
• Signal scanner: full-market technical signal detection with configurable score thresholds;
• Custom rule-based selection: AND-combined filtering over 70+ technical and asset fields;
• Strategy backtesting and optimization: built-in strategies with risk metrics and parameter sweeps;
• AI stock-picking assistant: ask questions in natural language; the system answers against real data via a read-only Text2SQL pipeline;
• Web dashboard: React + FastAPI, 10 tabs in total.

Keeping these lines in one project means data is ingested only once and every module reuses the same analytics library. The cost is that engineering boundaries must be kept clean, otherwise indicator computation, scanning logic, and the web service quickly couple into a tangle.

---

2. Overall Architecture: A Responsibility-Layered src/

The real project structure (excerpted from the repo root src/) is layered by responsibility, not a single giant script:

src/
├── agent/      # AI stock picking (read-only Text2SQL): llm / sqlsafety / schema / pipeline / portfolio / fallback
├── data/       # Data fetching & cleaning: fetcher(open-source market-data source) / stock_info / asset / turnover / sync_env
├── etl/        # ETL pipeline: ingest daily K-lines → compute 51 indicators
├── scanner/    # Signal scanning: signals / screener / monitor / accuracy / risk_alert / visualization
├── scorer/     # Ranking
├── strategy/   # Backtest / optimization / sector rotation / portfolio / risk control / market timing
├── web/        # FastAPI service + paper trading: api / api_cache / paper_trading
├── utils/      # Common utilities
└── main.py     # CLI entry

The data layer consists of three SQLite databases (all under data/):

Database	Role
stock_klines.db	Raw daily K-lines (source library), fetched by data/fetcher.py from an open-source market-data source
stock_analysis.db	Analysis library, ~7 million rows, 51 indicators, generated from the source library by make etl
asset_snapshot.db	Full-market market-cap / share-count / valuation snapshot, backing asset-based filtering

Separating the source library from the analysis library is an important design decision: raw quotes and derived indicators are decoupled, so when the cleaning logic or indicator definitions change, the analysis library can be safely recomputed without touching the source data.

The value of layering shows up in three places:

1. Indicators decoupled from data: indicator computation only reads DataFrames; the data source (local DB or online API) is managed uniformly by data/, so swapping sources doesn't touch indicator code.
2. Independent scanning logic: the signal scanner consumes indicator results through a unified interface; adding or replacing an indicator doesn't require changing the scanner.
3. Testability: ETL, scanning, and strategy can each be unit-tested independently.

---

3. Indicator System: Computing and Storing 51 Indicators

The 51 technical indicators are computed in src/etl/pipeline.py, implemented with pandas (e.g. _calculate_macd / _calculate_rsi / _calculate_boll / _calculate_kdj / _calculate_obv), rather than in a separate "indicator library" module. The indicators span trend, momentum, volatility, and volume dimensions, with typical implementations including MACD, RSI, KDJ, BOLL, and OBV.

Recomputing the full market is expensive, so the ETL uses a genuine incremental window: only recent data is recomputed, cutting write volume to roughly 1/6 of a full recompute. The output lands in stock_analysis.db, about 7 million rows per library, read uniformly by the scanning, backtesting, and web layers.

Note: the "category counts" of indicators vary under different classification schemes (one indicator may serve both trend and momentum judgments). This article deliberately avoids giving a mutually exclusive four-category tally to prevent distorted numbers; what matters more in practice is that indicators are configurable and results are reusable.

---

4. Signal Scanning: From Indicators to Trading Signals

Signal scanning explicitly defines signal types via the SignalType enum in src/scanner/signals.py. The real signal types include:

Category	Signal examples
Golden / death cross	MACD golden/death cross, KDJ golden/death cross, MA5 crossing above/below MA20
Overbought / oversold	RSI overbought/oversold, Williams %R overbought/oversold, CCI overbought/oversold
Breakout	Breakout above the upper Bollinger band, breakdown below the lower band, price breakout
Volume-price	OBV divergence, abnormal volume
Trend	Uptrend, downtrend, MA alignment, momentum
Candlestick pattern	Hammer, inverted hammer, bullish/bearish engulfing, morning/evening star

The scanner supports configurable score thresholds: each matched stock gets its signal type and a score, filterable and sortable by score. This complements the "custom rule-based selection" — the rule engine handles "explicit condition combinations" (e.g. market cap + P/E + volume ratio), while score-based scanning handles "composite signals triggered by indicators."

---

5. Custom Rule-Based Selection: AND Combinations over 70+ Fields

Beyond indicator signals, the system also offers filtering based on asset fields: covering 70+ technical and asset fields (market cap, free-float market cap, P/E, P/B, volume ratio, etc.), supporting multi-condition AND combinations to filter the full market. This line solves the "I know what hard conditions I want, but can't manually flip through the whole market" problem, and is a different approach from the score-based signals in Section 4.

---

6. Strategy Backtesting and Optimization

src/strategy/ is a collection of strategy-related capabilities: backtest.py provides built-in strategy backtesting, optimization.py and scripts/param_sweep.py do parameter sweeps, and portfolio.py / risk_control.py / benchmark.py handle portfolio and risk control, while market_timing.py / sector_rotation.py handle market timing and sector rotation.

The backtesting module can tally signal/strategy win rates, profit-loss ratios, and per-indicator contribution, and output them as reports. It must be emphasized: backtest results are for method validation, not a representation of future returns; signal effectiveness is highly dependent on market conditions, which is exactly why the system separates "timing" and "sector rotation" into their own modules.

---

7. AI Stock Picking: A Read-Only Text2SQL Agent

Imagine this: you simply ask, "Which stocks had a MACD golden cross in the last 20 days and an RSI below 40?" — the system translates that sentence into SQL on its own, runs it read-only against the real quote data, and returns results with a chart. You don't write a single line of SQL.

src/agent/ implements exactly this capability: a read-only Text2SQL pipeline (pipeline.py): natural-language question → generate SQL → read-only validation → execute → answer/visualize based on results, with self-correction on failure (up to 3 retries each for generation and execution). It replicates the method of work/harness/datapulse — SQL generation and final answering each have independent retry budgets. For the design details and trade-offs of this Text2SQL agent, see the dedicated datapulse article.

The safety boundary is guaranteed by three layers (see agent/sqlsafety.py and the README):

1. Read-only database: the analysis library is opened with mode=ro;
2. SQL whitelist validation: blocks write operations, dangerous functions, multiple statements, and excessively large LIMITs;
3. Result cap: at most 200 rows returned per query.

That is, even if the model errs, it can only err inside a "read-only, restricted, small-result-set" sandbox — it cannot touch the source data or perform writes. When LLM credentials are not configured, the other 9 tabs remain fully usable (the fallback module covers them).

The model is accessed through an OpenAI-compatible LLM API (configured via the environment variables LLM_BASE_URL / LLM_API_KEY / LLM_MODEL), with no specific vendor bound at runtime. The publicly hosted demo environment (https://stock-analyzer-demo.onrender.com) is completely isolated from the local production database — the demo database uses a set of placeholder stocks (DemoXX) to simulate quotes, carrying no real holding data.

---

8. Tech Stack and Engineering Trade-offs

Dimension	Choice
Runtime	Python 3.11+, dependencies managed with uv
Indicator computation	pandas + numpy (inlined in etl/pipeline.py)
Web backend	FastAPI (local :8001)
Web frontend	React + Vite + TypeScript (local :3000), charts via ECharts, red-up/green-down color scheme
Storage	SQLite (source / analysis / asset-snapshot libraries separated)

On "why pandas rather than polars / Rust": this project chose pandas, fundamentally as a trade-off for indicator implementation and iteration speed. The logic for 51 indicators is scattered across the various _calculate_* methods; pandas' ecosystem (TA-Lib, numpy) and readability make these easy to write and test. The timeliness of full-market scanning is met through incremental ETL + SQLite caching, not by swapping the compute kernel. For a toolkit whose goal is "research + maintainability," this combination is more pragmatic than pure Rust.

---

9. Data Quality and Boundaries

The ceiling of data quality determines the credibility of the indicators. The system splits "acquisition" and "analysis" into two libraries: daily data returned by the open-source market-data source first lands in stock_klines.db, then ETL cleans and adjusts it (adjustment for dividends/rights) to generate stock_analysis.db. The dual-library separation lets changes to the cleaning caliber be safely replayed, and avoids the source data being polluted by derived computations.

The asset-dimension snapshot (asset_snapshot.db) is stored separately, so needs like "filter by market cap / valuation" don't depend on real-time pulling, and scanning can run stably and reproducibly.

---

10. Current Status and Future Directions

Capabilities already in place: ETL + 51 indicators, signal scanning (configurable thresholds), custom rule-based selection (70+ fields), asset snapshot, strategy backtesting and optimization, market analysis (timing / breadth / sector rotation), paper trading, AI stock-picking assistant, and a web dashboard (10 tabs).

Directions to keep polishing:

1. Market environment recognition: automatically judge bull/bear/range-bound and adjust signal thresholds;
2. Signal decay model: different signals decay in effectiveness over time; introduce a decay factor;
3. Sector linkage analysis: introduce a sector dimension to filter out weak-sector stocks;
4. Real-time transformation: move from daily scanning to intraday monitoring.

---

Source Code Navigation

• src/etl/pipeline.py — ETL pipeline and 51-indicator computation (pandas)
• src/scanner/signals.py — signal type definitions and scanning logic
• src/strategy/backtest.py — backtesting module
• src/agent/pipeline.py — read-only Text2SQL AI stock-picking pipeline
• src/web/api.py — FastAPI service entry
• src/data/fetcher.py — market-data fetching

---

Project Repository

• GitHub repository: https://github.com/erishen/stock-analyzer
• Live Demo (Render): https://stock-analyzer-demo.onrender.com

---

⚠️ Disclaimer: This project is for technical learning and research only, and does not constitute investment advice. All technical indicators, signals, and AI responses are for reference only and their accuracy is not guaranteed. The stock market carries risks — please do not use this tool for real investment decisions.