Video analytics accuracy: why AI counts fail and how to prove it

Short answer: AI counts from video go wrong for a handful of predictable reasons: heavy compression that blurs people, occlusion when people overlap, trackers that lose or swap identities, and people lingering at the counting line so they are counted twice. You prove a count is accurate by having people count a sample of the same footage, matching every AI-counted crossing to a real one, and reporting precision and recall rather than just comparing totals.

Why do AI video counts go wrong?

A video counting system usually does three jobs, one after another:

1. Detect: find each person or vehicle in each frame and draw a box around it.
2. Track: link those boxes across frames so the system knows box A in frame 10 is the same person as box B in frame 11.
3. Count: decide when a tracked person has crossed a line or entered a zone, and add one to the count.

An error at any step flows through to the count. A missed detection becomes a missed person. A broken track becomes a new person. A shaky track near the line becomes extra counts.

The table below lists the common failure modes and what each does to the number.

How does low-bitrate CCTV footage affect accuracy?

Many CCTV systems record or stream at low bitrates to save storage and network capacity. The image you see on the monitor may look fine to a person, but the detector sees something much rougher.

Some simple arithmetic shows how tight the budget is. A camera streaming at 300 kilobits per second at 15 frames per second has, on average, 20 kilobits per frame. That is about 2.5 kilobytes to describe an entire picture. At 1280 × 720 resolution, that works out to roughly 0.02 bits per pixel per frame. The encoder copes by throwing away fine detail and reusing parts of earlier frames, which is why edges smear and fast movement turns into blocks.

What does that do to AI detection? A 2022 study encoded 50 annotated surveillance videos at five compression levels and ran a widely used open-source object detector on each. The authors found detection was "generally robust to moderate levels of compression," but "degrades appreciably at higher compression levels," and the losses were worst in complex scenes with poor lighting and fast-moving targets. Retraining the detector on compressed footage helped only a little: up to a 1% improvement in F1 score on the most compressed video.

The practical lessons:

• Compression sets a ceiling. If the footage is badly compressed and there is no better source, no model will fully recover the lost detail. Accuracy work then shifts to tracking, counting logic and human review.
• Check the actual stream, not the camera spec. A 4K camera recording at a low bitrate is not giving you 4K detail.
• Test at the conditions you will run in. A demo on clean, high-bitrate footage does not tell you how the system performs on your night-time feed.
• Small and fast targets suffer first. People far from the camera and vehicles at speed are the most affected.

What is an ID switch, and why does it double-count?

A tracker gives each person an ID and tries to keep it the same while they are in view. An ID switch happens when that identity changes for the same real person, or two people swap IDs.

The MOT16 tracking benchmark counts an identity switch as an error alongside missed detections and false positives. It combines all three in the widely used MOTA score:

MOTA = 1 − (missed + false positives + ID switches) ÷ number of real objects

In a counting system, ID switches cause specific problems:

• A person walks behind a pillar, the track is lost, and they reappear with a new ID. If the new track then crosses the line, they can be counted a second time.
• Two people cross paths and swap IDs. One track may appear to cross the line twice while the other never does.

MOTA is not the only measure. The authors of HOTA, a later metric, argued that earlier metrics tended to weight either detection or association too heavily, and designed HOTA to balance detection, association and localisation. For counting, though, what matters most is whether each crossing was counted once, which the next sections cover.

How do line-crossing counts produce double counts?

Most counting systems draw a virtual line across a doorway or road and count a person each time their track crosses it.

The weak point is the line itself. People stop at doorways, step back, hold doors and chat. Even when they stand still, the box around them wobbles a few pixels each frame, especially on compressed video. A naive rule of "count every crossing" counts each wobble.

Common fixes:

1. Use a buffer zone. Count a crossing only when a track enters from one side and fully exits the other side of a band, not when it touches a single line.
2. Respect direction. Count "in" and "out" separately, and cancel a crossing if the same track reverses within a short time.
3. Require a minimum track length. Ignore tracks that exist for only a few frames; they are often false detections.
4. Place the line where people keep moving. Not right at a door where they stop, and not where they are hidden by a counter or shelf.
5. Choose camera angles that reduce overlap. A higher, more top-down view separates people in crowds better than a flat, eye-level view.

How do you measure counting accuracy against ground truth?

You cannot know how accurate a count is without a trusted reference. That reference is ground truth: people carefully counting the same footage by hand, often with the ability to pause and rewind.

Step 1: Pick a fair sample

Choose clips that reflect real conditions: each camera, day and night, quiet and peak hours, rain or glare if it occurs. A sample of only easy daytime footage will flatter the system.

Step 2: Count by hand, crossing by crossing

Reviewers record each crossing with its time and direction, not just a total. Where reviewers disagree, a second person settles it.

Step 3: Match AI crossings to real crossings

For each AI-counted crossing, look for a real crossing at the same line, in the same direction, within a small time window. Each AI count is then one of:

• True positive (TP): matches a real crossing.
• False positive (FP): no real crossing matches it (a double count or a phantom).
• False negative (FN): a real crossing that the AI missed.

Step 4: Calculate precision and recall

• Precision = TP ÷ (TP + FP). Of the crossings the AI counted, how many were real?
• Recall = TP ÷ (TP + FN). Of the real crossings, how many did the AI catch?

Why totals are not enough

Here is an illustration of why the total can mislead.

The total looks almost perfect: only 2% off. But 20 individual crossings were wrong. The misses and double counts cancelled each other out. On a different day, with a different mix of conditions, they might not cancel, and the total could swing much further.

So always report precision and recall, by camera and by condition, alongside the total.

Where does human review fit in?

Even with good models and careful line placement, some cases will be uncertain. Human review makes those cases accurate instead of guessing.

A practical review process:

• Flag the uncertain cases automatically. Good signals include low detection confidence, tracks that break and restart near the line, crowded moments, and disagreement between independent methods.
• Show reviewers the evidence. A short clip of each flagged crossing, with the track drawn on it, so they can confirm or reject it in seconds.
• Audit a random sample of confident cases too. Reviewing only flagged cases tells you nothing about the errors the system was confident about.
• Feed the results back. Reviewer decisions show which cameras, times and conditions cause errors, which tells you where to move a line, adjust a camera or retrain.

A checklist for proving your counts

1. Confirm the real bitrate, resolution and frame rate of each feed.
2. Place counting zones where people keep moving, with a buffer between entry and exit lines.
3. Build a ground-truth sample across cameras and conditions.
4. Match crossings one by one and report precision and recall, not just totals.
5. Route uncertain crossings to human review, and audit a random sample of the rest.
6. Publish the accuracy figures with the count, so anyone using the number knows how far to trust it.

How CloudOEngine helps

CloudOEngine's Vision AI is built around proving counts, not just producing them.

Each count is produced by two independent trackers. Where they agree, the crossing is counted. Where they disagree, the case goes to a human review queue, where your own staff check the clip and decide. That focuses human effort on the crossings most likely to be wrong, and the final decision on every disputed case rests with people who know the site.

When a count is finalised, it ships with a published accuracy report, and every counted crossing is shown, so anyone who questions a number can see exactly which crossings make it up. That is the same principle this article argues for: a count is only as useful as the evidence behind it.

Vision AI sits on the same platform as CloudOEngine's data connections, pipelines, AI runtime and live video, with one permission model and one audit trail. Camera streams connect alongside your databases and files, and governance is included: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets.

CloudOEngine is built in Bengaluru, India.

Book a demo · See Vision AI

Frequently asked questions

Why are AI people counts from CCTV wrong?

The usual causes are heavy video compression that blurs small or fast-moving people, occlusion when people overlap, tracker ID switches that make one person look like two, and people hovering near the counting line so they are counted more than once.

How do you measure the accuracy of an AI counting system?

Have people count the same footage by hand to create ground truth, then match each AI-counted crossing to a real one. From the matches you calculate precision (how many AI counts were real) and recall (how many real crossings the AI caught).

Is comparing the AI total with the manual total enough?

No. Missed people and double counts cancel each other out, so a total can look right while many individual counts are wrong. Event-level matching shows the real error rate.

Does low-bitrate CCTV footage reduce AI accuracy?

Yes, at high compression. A 2022 study of surveillance video found detection held up under moderate compression but degraded noticeably at higher compression, especially in poor light and with fast-moving targets.

What is an ID switch in video tracking?

An ID switch happens when a tracker swaps or loses the identity of a person between frames, for example after they pass behind a pillar. In counting, it can make one person look like two people or let two people be counted as one.

Sources

1. Impact of Video Compression on the Performance of Object Detection Systems for Surveillance Applications (O'Byrne et al., 2022)
2. MOT16: A Benchmark for Multi-Object Tracking (Milan et al., 2016)
3. HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking (Luiten et al., 2020)

#video analytics #people counting #computer vision #accuracy #cctv