HOW THE RADAR
COUNTS THINGS.

WHAT THE RADAR LISTENS TO

Permitted public technical-problem posts, collected through official or public interfaces: GitHub issues and discussions, and Hacker News. Communities whose terms forbid automated access aren’t collected automatically. A person may add a specific public post by hand. It’s a limited, targeted sample, not the whole internet.

PUBLIC SIGNAL

A post by a person describing a concrete technical problem they are experiencing. Posts written by AI agents or automation are excluded from the public figures and counted separately. A public signal is not a verified bug.

SAID FIXED / STILL BROKEN

Among qualifying human public AI repair-attempt reports in the observed dataset, the count and share where the poster subsequently reported that the original behavior remained broken or another regression appeared.

• Qualifying means a person says an AI tool or agent attempted a fix for their specific problem. Using AI somewhere in a project doesn’t count, and general frustration with AI doesn’t count.
• The count is always shown with its denominator, and the own/client-project subset is kept separate from reports about a tool vendor’s own product.
• It is not the percentage of AI fixes that fail. People mostly post when something goes wrong, so this describes the posts we observed and says nothing about how often AI repairs work.

RECURRING PATTERNS

Each signal is assigned to one pattern from a fixed list of symptoms. A pattern recurs when several independent people report it within the window. A pattern is a group of similar symptoms, not a root cause. Pattern counts are not shown until there is enough data.

REPRODUCED FAILURES AND VERIFIED FIXES

These come only from actual Proof First cases, never from the Radar. A case counts as verified only with BEFORE = FAIL, SAME TEST LOCKED, repair, the same test rerun, and AFTER = PASS, with the evidence saved. Controlled demonstrations on sample apps, such as the video on the home page, are never counted as customer cases. A model saying “fixed” isn’t verification.

HOW IT’S CHECKED

An automated classifier (Jev) assigns bounded labels to each post. It doesn’t repair software. We compared its answers with a reviewed sample of 52 real cases: people decided every disagreement and a random spot check, with an AI giving an independent second opinion. It is a small test, not a guarantee, and the individual counts shown on the home page are made automatically and are not reviewed one by one. Counts change as new reports are collected.

WHAT THE RADAR KEEPS

Every post the Radar collects is public. For each one it keeps the post’s text, link and date. It does not store the writer’s username as such; it keeps a one-way code, used only to count distinct writers. The link and the text can still contain public names, such as a repository owner or an @mention. These are people who posted in public, not Proof First customers. The Radar never emails or privately messages them; any reply stays public, in the same thread. Posts are labelled automatically, using a cloud classifier (Jev, from TypeSafe) that receives the public post text. The labels are not reviewed one by one, and the Radar is a limited sample, not a complete record of anyone or anything.

The data is kept while the research is active. If you wrote a post the Radar holds and want it removed, email christopherarivero@gmail.com with the link to the post and I’ll delete it. Everything else about what Proof First collects is on the privacy page.

KNOWN LIMITATIONS

• Only some public sources are covered, and the people most likely to post publicly are not typical of all builders.
• Many collected reports come from developers of AI tools rather than from people running their own projects, which is why those are separated.
• Classification can be wrong. That’s the reason for the human check.
• Counts describe the observed dataset in a time window. They don’t describe all software problems.