What is a human activity recognition dataset?

A human activity recognition (HAR) dataset is a collection of video clips, skeleton sequences, or sensor traces labeled with the activity a person is performing — walking, falling, fighting, reaching, loitering. Vision-based HAR datasets pair each clip with an activity class and, ideally, temporal boundaries marking when the activity starts and ends.

What is the difference between HAR and action recognition?

Action recognition classifies short atomic movements from a few seconds of video, such as a wave or a jump. Human activity recognition is broader and covers longer, compound behaviors that unfold over many seconds — loitering, shoplifting, a fall event with its aftermath — and usually requires temporal segmentation, not just a single clip label.

Which human activity recognition datasets are most used?

The common public benchmarks are UCF101, HMDB51, Kinetics-400/700, NTU RGB+D for skeleton-based HAR, and UCF-Crime for anomalous surveillance activity. They are excellent for benchmarking but weak for production safety and security systems, because rare and dangerous activities are under-represented and the camera geometry rarely matches a real deployment.

How is a synthetic HAR dataset labeled?

Synthetic HAR data is labeled at render time. The engine already knows each character's skeleton, action state, and position, so it emits per-frame keypoints, bounding boxes, and clip-level activity classes automatically — with exact temporal boundaries and no inter-annotator disagreement.

Is synthetic HAR data GDPR compliant?

Yes. Every person in a synthetic HAR clip is computer-generated, so no real biometric data is captured or processed. That removes the consent, retention, and lawful-basis obligations that apply when training activity models on real CCTV footage under GDPR and the EU AI Act.

Human Activity Recognition Datasets

What a human activity recognition dataset is

A human activity recognition dataset is a set of clips — or skeleton sequences, or sensor traces — where each sample carries a label describing what the person is doing. In computer vision the label is usually an activity class plus a temporal window: fall, 3.2s–5.8s. That temporal boundary is the part teams underestimate. Getting the class right is easy; getting the onset right is what makes an alarm useful.

The clip below is one of ours. A synthetic person collapses from cardiac arrest on a corridor camera. Nobody was filmed, nobody was hurt, and every frame ships with a labeled skeleton and an activity class with exact start and end frames.

Synthetic HAR sample: medical collapse captured from a fixed indoor camera, annotated per frame at render time.

HAR vs action recognition

People use the terms interchangeably and then get confused about why their model underperforms. The distinction is practical:

Action recognition — short atomic movements, one class per trimmed clip. A wave, a kick, a jump. Benchmarks: UCF101, HMDB51, Kinetics.
Human activity recognition — longer, compound behavior that unfolds over time and often needs temporal segmentation on untrimmed footage. Loitering, shoplifting, a fall and its aftermath.
Video anomaly detection — the unsupervised cousin, where "abnormal" is anything outside learned normal behavior. Useful when you cannot enumerate your classes.

If your product raises an alert, you are almost certainly doing HAR on untrimmed video, not clean action recognition. Read the wider framing in our video datasets for computer vision guide.

How HAR data gets labeled

Three approaches dominate, and they cost wildly different amounts:

RGB clip labels. A human watches and tags. Cheapest per clip, weakest temporal precision, and painful at scale.
Skeleton-based. A pose estimator extracts 17 keypoints per frame and the classifier runs on joint trajectories. Robust to lighting and clothing, and the standard for NTU RGB+D-style work — but it inherits every pose-estimation error.
Render-time ground truth. In synthetic pipelines the engine already knows the skeleton, the action state, and the exact frame the activity begins. Labels come out perfect and free.

Field note

On our own fall-detection work, the single biggest driver of false alarms was not the fall class — it was the negatives. Sitting down quickly, bending to pick something up, and lying down deliberately all look like a fall to an under-trained model. Rendering a few thousand lookalike negatives moved precision more than doubling the positives did.

Where the public benchmarks fall short

UCF101, Kinetics, NTU RGB+D and UCF-Crime are great for publishing a number. They are poor for shipping a safety system. The activities you actually care about — a collapse, a weapon draw, a hold-up — appear in a handful of clips, filmed from handheld or web video, at framing that looks nothing like a ceiling-mounted 1080p camera in a corridor. The mismatch in camera height, lens, compression, and lighting is usually a larger accuracy tax than model choice. We break the tradeoffs down in synthetic data vs real data.

The activity clusters with real demand

These are the HAR classes buyers ask us for, each backed by a live Simuletic dataset or write-up.

Falls and medical collapse

Slips, trips, syncope, and cardiac collapse in eldercare, hospitals, and workplaces — the highest-value HAR class and the hardest to collect.

Fall detection dataset

Aggression and fighting

Shoving, punching, weapon draws, and pre-attack posture. Needs both positives and lookalike negatives to keep false alarms down.

Aggression dataset

Retail loss behaviors

Concealment, sweep-and-grab, and loitering near high-value shelves, filmed from realistic overhead store cameras.

Shoplifting dataset

Hold-up and robbery

Weapon presentation, forced withdrawal, and hands-up compliance behavior around ATMs and counters.

ATM robbery dataset

Driver and operator state

Drowsiness, microsleep, head-nod, and distraction sequences for in-cabin and control-room monitoring.

Drowsiness dataset

Pedestrian behavior attributes

Gait, carried items, and clothing attributes that give downstream HAR models context beyond the raw action class.

Pedestrian attribute dataset

Building a HAR dataset that survives deployment

Fix the camera first. Mount height, tilt, lens, and resolution should match the deployment before a single clip is rendered.
Enumerate the negatives. For every positive class, list the three behaviors that look like it and generate them too.
Vary the nuisance factors. Lighting, clothing, body type, occluding furniture, crowd density. This is where synthetic pipelines quietly win.
Label temporally, not just categorically. Onset accuracy determines alert latency.
Validate on real footage. Train mostly synthetic, hold out a small real set for honest evaluation.

How Simuletic ships HAR data

Every synthetic activity dataset we deliver includes per-frame YOLO bounding boxes, optional 17-keypoint COCO skeletons, and clip-level activity labels with exact temporal boundaries. You can browse the current library on the datasets page, or contact us about a custom scenario dataset.

Need an activity class nobody has filmed?

Send us the behavior, the camera geometry, and the label format. We render synthetic HAR clips with per-frame annotations, usually in days rather than months.

Request a custom HAR dataset Browse all datasets CCTV anomaly guide