# Hybrid log watching on a DGX Spark: regex counters plus Nemotron 3 Nano Omni on a real 1M-lines/hour SIP proxy log

> A DGX Spark recipe by Greg Alton: the counters-plus-model pattern tested on a live SIP proxy log at about 1.17M lines an hour, with honest numbers, weak spots, and fixes.

Level: Advanced
Tool: DGX Spark
Category: Building
Time: About 25 minutes
Checked: 2026-10-16
Tools: [vLLM](https://docs.vllm.ai), [vLLM structured outputs](https://docs.vllm.ai/en/latest/features/structured_outputs.html), [NVIDIA DGX Spark playbooks](https://build.nvidia.com/spark)
HTML page: https://easyaihowtos.com/guides/hybrid-log-watching-nemotron-nano-omni-dgx-spark

AI engineering is just process engineering. *— [Greg Alton](https://www.linkedin.com/in/greg-alton-ios)*

My last recipe ended with a promise: try a smaller model, and try a real busy log. The synthetic test said the hybrid pattern works: cheap counters read every line, and a model explains what tripped. But synthetic logs are polite. Real ones aren't.

So I switched the Spark from Nemotron 3.5 Super VL to the smaller **Nemotron 3 Nano Omni**, pointed the watcher at a live SIP proxy log moving at more than a million lines an hour, and let it run for 15 minutes. I bought my DGX Spark to learn by doing. Here's what I learned, including the parts that didn't work.

## What you need

- A DGX Spark (or another GPU box) serving a model through an OpenAI-compatible API, such as vLLM
- Read-only SSH access to a log you're allowed to read
- Python 3 with the requests package
- The counter code from the earlier recipe, Can a vision model watch tail -f?

## Steps

### 1. The setup

In the [first recipe](/guides/can-a-vision-model-watch-tail-f-dgx-spark), a vision model couldn't keep up with a fast-scrolling terminal, but counters plus a model could. That was on synthetic logs. This time I pointed the same pattern at **a live SIP proxy log, read-only**.

- **The stream:** `ssh ... tail -F` on the log, piped to the Spark. Nothing was written on the server, and the raw lines stayed on the Spark.
- **The counters:** the regex watcher from the first recipe, running on every line.
- **The model:** Nemotron 3 Nano Omni (NVFP4), served by vLLM on the Spark. I swapped it in for Nemotron 3.5 Super VL because the first test showed the model was the bottleneck.
- **The trigger:** when a counter trips (and every N seconds for context), send the model a recent sample and ask for a JSON verdict.
- **The run:** 15 minutes.

### 2. The results

| Measure | Result |
|---|---|
| Lines in 15 minutes | about 292k |
| Average rate | 324 lines/s (about 1.17M lines/hour) |
| Watcher CPU | about 0.6% (1.2% for the whole system) |
| Counter trips | 12 |
| Time from trip to model verdict | 0.75–2.35 s |
| Image call, Nano Omni | 1.12 s average (median 0.72 s, p90 1.61 s) |
| Image call, Super VL (earlier) | about 2.4–4.1 s |
| Text call, Nano Omni vs Super VL | 0.72 s vs 1.16 s |
| GPU use | 8% average, 96% peak during calls |
| Invalid JSON replies | 12% |

The counters read every line of a real 1M-plus-lines-an-hour log for less than 1% CPU. The model answered every trip within about 2.4 seconds. The smaller model was roughly 2–3 times faster on images, and the GPU sat mostly idle between calls.

### 3. The weak spots

It wasn't all good news, and this is the useful part.

- **The error counter was mostly noise.** It matched 1,345 lines. About 65% were harmless debug lines that just contain the word "error." Only about 35 were true error-level lines, plus 6 warnings. A word match isn't a severity.
- **The model mostly echoed the counters.** Ask it "these counters tripped, what's going on?" and it tends to say the counters tripped. That adds little.
- **12% of replies were invalid JSON.** Each one needs a retry or a fallback, or an alert gets dropped.
- **The log is big.** It rotates daily at about 3–4 GB uncompressed per day, with no size cap. Anything that re-reads whole files will hurt, so stream new lines only.

### 4. Fix 1: level-aware regex

Match on the log level field, not on a word anywhere in the line. Adjust the pattern to your log's format:
```python
import re

# Match the level token near the start of the line, not "error" anywhere.
LEVEL = re.compile(r"^\S+\s+\S+\s+\[?(DEBUG|INFO|NOTICE|WARN(?:ING)?|ERROR|CRIT(?:ICAL)?|FATAL)\]?\b", re.I)

def level_of(line):
    m = LEVEL.search(line)
    return m.group(1).upper() if m else None

# Count ERROR and above for alerts. Ignore DEBUG entirely, even if it says "error".
ALERT_LEVELS = {"ERROR", "CRIT", "CRITICAL", "FATAL"}
```
In this run, that change alone would have turned 1,345 "errors" into about 35 real ones.

### 5. Fix 2: guided JSON in vLLM

Don't hope for JSON; constrain it. vLLM's OpenAI-compatible server supports structured outputs, so you can pass a JSON schema and the decoder can only produce valid output:
```python
import requests

SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string", "enum": ["ignore", "watch", "page"]},
        "likely_cause": {"type": "string"},
        "new_pattern": {"type": "string"},
        "next_check": {"type": "string"},
    },
    "required": ["verdict", "likely_cause", "new_pattern", "next_check"],
}

r = requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "your-model",
    "messages": [{"role": "user", "content": prompt}],
    "response_format": {"type": "json_schema",
                        "json_schema": {"name": "verdict", "schema": SCHEMA}},
    "temperature": 0, "max_tokens": 300,
}, timeout=60)
```
Check the [vLLM structured outputs docs](https://docs.vllm.ai/en/latest/features/structured_outputs.html) for the options your version supports. Keep a fallback anyway: the counter's own alert should go out even if the model fails.

### 6. Fix 3: ask for what counters can't tell you

The counters already know *how many*. Don't ask the model to recount. Ask for what needs judgment:

- **Is this new?** Does the sample look different from normal traffic?
- **What's the likely cause?** For example, one client retrying, or many failing at once.
- **What should a human check next?**
- **Is there a pattern worth a new counter?** The best output is a regex you can add, so the cheap layer gets smarter over time.

Send the counter totals as context and say plainly: "Do not repeat these numbers."

### 7. The takeaway

**The counters do the heavy lifting, and the model adds context.** The counters read every line, cost almost nothing, and never miss. The model is fast enough now (about a second a call on Nano Omni) to explain every trip, but it only earns its place when you ask it questions the counters can't answer.

Next I'll rerun this with level-aware counters, guided JSON, and the new prompt, and see if the model starts adding real insight. Follow the [DGX Spark section](/dgx-spark) for the next recipe.

## Tips

- Stream with tail -F, not tail -f, so the watcher survives daily log rotation.
- Keep it read-only. The server only runs tail; all parsing and the model run on the Spark.
- Keep logs local. Raw lines, addresses, and numbers never leave your network.
- Pick the smallest model that answers fast. The model was idle most of the run, so speed per call matters more than size.

## FAQ

### Why not let the model read every line?

At about 324 lines a second, that's far more than a model can read in real time. Regex counters handled it for under 1% CPU.

### Text or screenshot for the sample?

Text when you have it. Text calls were faster (0.72 s on Nano Omni) and can't be misread. Use screenshots only when a screen is all you have.

### Was the model useful at all?

It was fast and reliable about responding, but it mostly repeated the counters. That's a prompt problem, not a speed problem, and it's what Fix 3 is for.

---

This guide is from Easy AI How-To (https://easyaihowtos.com). It is independent of the tools named above. Screens and free-plan limits change, so check the date before you rely on a button name.