# Real Data Integration Framework

**Goal:** Build the dashboard from what we actually have in collected data, with a framework to handle new data as it arrives.

**Status:** Evidence collection test complete. Real signals extracted and ready for dashboard integration.

---

## Current State: What We Found

### Edition Zero Collection (Mar 14, 20:12-20:50 UTC)
- **Total articles collected:** 9,116
- **Real signals detected:** 70
- **Pillar with signals:** Evidence (Investigators Departed)
- **Query triggering signals:** "FBI Iran domestic", "federal personnel Iran"

### Sample Real Signal
```
Title: "US Faces Elevated Terrorism Threats Against Backdrop Of Iran War And Cuts At FBI"
Source: etvbharat.com
Date: 2026-03-14
URL: https://www.etvbharat.com/en/international/...
Query that triggered: "FBI Iran domestic"
```

**Why this triggered:** Article mentions "cuts at FBI" (investigators departing) in context of Iran war.

---

## The Framework: Data Flow

### Step 1: Collection (24/7)
```
GDELT / NewsData / NewsAPI / Reddit
         ↓
   9,000-10,000 articles/day
         ↓
   /root/.openclaw/workspace/memory/signal_results/
         ↓
   497+ JSON files (Edition Zero example)
```

### Step 2: Analysis (3x daily)
```
Edition Zero files
         ↓
   signal_analyzer_v4.py runs
         ↓
   Deduplicates by outlet
   Multi-API coordination check
   Operational pattern detection
         ↓
   /root/.openclaw/workspace/memory/analysis_reports/
         ↓
   ZERO_EDITION_v4_20260315_060000.txt
```

### Step 3: Signal Extraction (New!)
```
Analysis output + Raw articles
         ↓
   extract_real_signals.py
         ↓
   Match article text to signal patterns
   Extract URLs + titles + metadata
   Deduplicate within indicator
   Take top N per indicator
         ↓
   real_signals_from_data.json
   {
     "evidence": {
       "Investigators Departed": [
         { title, url, source, date, snippet }
       ]
     }
   }
```

### Step 4: Dashboard Integration
```
real_signals_from_data.json
         ↓
   Update modal-signal-sources.js
   Replace demo data with real data
   OR supplement demo with real
         ↓
   Dashboard refreshes
         ↓
   User clicks indicator
   Sees REAL articles from collected data
   Clicks to verify in original source
```

---

## Implementation: 3 Phases

### PHASE 1: Extract from Edition Zero (NOW)
- ✅ Run `extract_real_signals.py` on 497 files
- ✅ Get list of real articles + sources
- ✅ Generate `real_signals_from_data.json`
- **Output:** Real signal data from collected articles

### PHASE 2: Dashboard Integration (NEXT)
- Convert `real_signals_from_data.json` → `modal-signal-sources.js` format
- Decide: Replace demo data OR supplement?
- Deploy to dashboard
- Test clicking articles

### PHASE 3: Automation (Tomorrow & Beyond)
- When `signal_analyzer_v4.py` runs at 06:00 UTC:
  1. Finishes analysis
  2. Auto-runs `extract_real_signals.py` on new data
  3. Updates `real_signals_from_data.json`
  4. Dashboard auto-refreshes with latest signals
  5. User sees latest articles from TODAY's collection

---

## The Signal Extraction Algorithm

### Pattern Matching
```python
SIGNAL_PATTERNS = {
    'evidence': {
        'Investigators Departed': [
            r'investigator.*quit|investigator.*resign|fbi.*depart',
            r'investigation.*close|investigator.*reassign'
        ]
    }
}
```

### For each article:
1. Extract title + URL + source + date
2. Search title for pattern matches
3. If match found → Add to indicator's source list
4. Deduplicate (same URL only once)
5. Limit to top 3 sources per indicator (most relevant)

### Output Format
```json
{
  "pillar_id": {
    "Indicator Name": [
      {
        "title": "Article title (first 80 chars)",
        "url": "https://...",
        "source": "domain.com",
        "date": "2026-03-14",
        "snippet": "Query that triggered this"
      }
    ]
  }
}
```

---

## Integration Points

### Current Real Data (Edition Zero)
- **Evidence → Investigators Departed:** 70 real articles mentioning "FBI cuts"
- Can populate this indicator with real articles from our collection

### Incoming Data (Tomorrow onwards)
- 06:00 UTC: Edition Zero v4 runs → 9,000-10,000 new articles
- Auto-extraction finds matching patterns
- Dashboard updates automatically
- User sees fresh signals from today's collection

---

## Decision Points

### Q1: Replace Demo Data or Supplement?
- **OPTION A (Replace):** Remove all demo data, show ONLY real signals
  - Pro: Pure real data, no confusion
  - Con: Only 70 signals initially (vs 18 demonstrated)
  - Best for: Production mode

- **OPTION B (Supplement):** Keep demo for other pillars, show real for Evidence
  - Pro: Shows what all pillars will look like when data comes
  - Con: Mixed demo + real could be confusing
  - Best for: Development/demo mode

- **OPTION C (Hybrid):** Demo for indicators with no data, real for those with data
  - Pro: Realistic + complete framework
  - Con: Some pillars empty initially
  - Best for: Transparent development

### Q2: Per-Pillar Updates or Full Refresh?
- **OPTION A:** Entire modal regenerates each run
- **OPTION B:** Only update indicators that have new signals
- **RECOMMENDATION:** Full refresh for consistency

### Q3: Manual or Automated Extraction?
- **OPTION A:** Manual: I extract, you review, we deploy
- **OPTION B:** Automated: Cron job runs extraction → auto-updates dashboard
- **RECOMMENDATION:** Start manual, move to automated after QA

---

## Timeline

### TODAY (Mar 15, 00:26 UTC)
- ✅ Extract real signals from Edition Zero collection
- ✅ Generate real_signals_from_data.json
- Create converter: JSON → modal-signal-sources.js format
- Decide on demo vs real vs hybrid approach

### TOMORROW (Mar 15, 06:00 UTC)
- Edition Zero v4 runs
- 9,000-10,000 new articles collected
- Auto-extract signals
- Dashboard updates with latest data
- 14:00 UTC: Second update
- 22:00 UTC: Third update

### ONGOING
- Each analysis run → auto-extraction
- Dashboard always shows latest signals
- Real articles from real collection
- Complete transparency

---

## Data Validation Checklist

Before deploying real signals to dashboard:
- [ ] URLs are valid (no 404s)
- [ ] Article titles are coherent
- [ ] Source domains are real news outlets
- [ ] Dates are in correct format (YYYY-MM-DD)
- [ ] Pattern matching is accurate (not false positives)
- [ ] Deduplication removes duplicates
- [ ] Top-N selection works (3 per indicator)
- [ ] JSON structure matches dashboard expectations

---

## Success Criteria

✅ Dashboard shows signals sourced from actual collected data  
✅ Each active indicator has real article links  
✅ Articles are from real, named news outlets  
✅ User can click and verify in original source  
✅ Framework handles incoming data automatically  
✅ No demo/fake data mixed with real  
✅ Complete transparency on data origin  

---

## Files Involved

### Input
- `/root/.openclaw/workspace/memory/signal_results/` — Raw collected data (497+ files)

### Processing
- `/root/.openclaw/workspace/extract_real_signals.py` — Signal extraction script
- `/root/.openclaw/workspace/signal_analyzer_v4.py` — Analysis engine

### Output (Current)
- `/root/.openclaw/workspace/real_signals_from_data.json` — Real signals from Edition Zero

### Output (Dashboard)
- `/home/adminmrk/htdocs/sigint/modal-signal-sources.js` — What dashboard displays
- `/home/adminmrk/htdocs/sigint/index.html` — Dashboard frontend

---

## Next Action

**To move forward, we need to decide:**

1. **Data source preference:** Real only? Demo + real? Hybrid?
2. **Deployment timing:** Deploy real signals now, or wait for fuller picture tomorrow?
3. **Automation preference:** Manual extraction then review, or auto-run after QA?

Currently we have:
- ✅ **70 real signals** from Edition Zero (Investigators Departed / FBI cuts)
- ✅ **Framework** for extraction
- ✅ **Real articles** ready to deploy

What's your preference?
