Can a 4-billion-parameter model run real research on a laptop? Yes, it can. Does its output hold up under scrutiny? That’s the interesting question, and a new paper (arXiv:2607.12257) has some sobering answers.
The headline finding is this: a small on-device research agent can search a corpus, read sources, and write a cited brief. It works. But whether its citations are actually faithful — whether the source it cites actually supports the claim it makes — depends on something specific and fixable.
Two things, not one
The paper makes a crucial distinction that most AI research glosses over. There are two separate questions here. First, does the cited source support the claim? That’s “cited claim faithfulness”. Second, did the agent cite the right sources at all? That’s “trustworthy coverage”. These are different problems driven by different factors.
Faithfulness turns out to be about exposure. If the model sees more of each source — the paper tested 400 characters versus 1500 — faithfulness climbs from 0.37-0.45 up to 0.58 regardless of whether the source is actually correct. More text in, better citations out. The extra reading costs about 235 output tokens. Cheap fix.
Coverage is harder. Trustworthy coverage stayed around 0.22 on retrieved sources no matter how much context the model saw. The reason is that the retrieval step only finds about 40% of the relevant sources, and no amount of context expansion can fix what wasn’t retrieved in the first place. If the right paper isn’t in the model’s working set, it can’t cite it — no matter how faithfully it reads what it has.
The practical take
If you’re building or using on-device research agents — and with 24GB laptops that’s increasingly viable — the fix order is clear. First, make sure the model reads enough of each source. That’s cheap, easy, and lifts faithfulness by a quarter. Second, fix your retrieval. Better recall is the only lever for coverage, and 40% recall means three out of five relevant sources are never even considered.
The study is small — one 4B model, one setup. Different models might behave differently. But the separation of faithfulness from coverage is a useful lens regardless. Next time someone pitches you an AI research agent that “cites everything accurately”, ask them which number they measured.
Source: “On-Device Deep Research at 4B” (arXiv:2607.12257)