Why Sound Inspection on the Line Is Hard
Sound is often the earliest sign of an assembly or part problem — yet handing that judgment to a machine reliably is harder than it looks.
Loose assemblies, burrs and bearing faults show up in the sound while every dimension is still in tolerance. Plants have long wanted listening as a formal inspection step, but three obstacles get in the way.
Relying on a seasoned inspector
A seasoned inspector hears a great deal, but the criterion differs by person and drifts with fatigue. It also leaves no trace, so it can be neither reviewed nor copied to the next line.
Relying on level and spectrum thresholds
Thresholds only catch what gets louder. Much of what goes wrong keeps its loudness and changes timbre: a faint rubbing texture, a periodic click.
Teaching an AI from labelled defects
Classification assumes enough defects of enough kinds. The line is the opposite: good parts dominate, defects are rare, and by the time samples suffice the line has moved on.
| What happens on the line | Why conventional methods struggle | What the algorithm must do |
|---|---|---|
| What to listen for is not obvious in the first place | Indicators must be thought up by people, so gaps remain | The model learns from data what to attend to |
| Defects are rare, and new kinds appear | Learning needs defect samples first | Build a criterion without labels |
| Same loudness, different timbre | Aggregate indicators cannot tell | Discriminate on timbre and texture |
| Background and incoming lots drift | Fixed thresholds go out of alignment | The criterion moves with the line |
| Every line has its own character | One setup cannot suit all | Each line picks what suits it |
Not more hand-written rules, but a deep learning system that learns for itself
It stays usable where defect samples are scarce and the line keeps drifting.
Three Design Principles
Against those three obstacles, the engine holds to three principles.
The three principles decide every method described later: what the model learns from without samples, how a judgment already in use survives new capability, and why nothing is compressed into one indicator.
Learn Without Defect Labels: Self-Supervised
The model sets its own task and checks its own answer, learning a representation of normal sound — its own internal description — from unlabelled audio: how energy moves, where resonances sit, how a strike decays. The criterion is learned, not written by hand.
No defect samples neededKeep the Pre-Trained Deep Listening Model Fixed; Extend How It Is Read
Once its version is fixed, the lower model's weights are never changed; a new line only means re-fitting the readouts and thresholds above it. It is not replacing the person with the good ear, but asking them to describe what they heard in another way.
No base retrainingOne Sound, Several Complementary Views
Some faults are obvious in the overall timbre, some appear only in fine temporal texture, some only when the sound is read as an ear would feel it. One indicator would let them dilute each other, so each view judges independently before the judgments are merged.
Complementary, not compressed- The existing baseline stays valid, and a new readout can run in record-only mode until its behaviour is confirmed.
- Any extension can be switched off to return to the previous state, and since the base layer is untouched, any difference comes from what was added.
A few terms used in this manual
Hearing Clearly: Several Scales at Once
A sound must first be heard clearly before it can be judged. This section explains how the engine listens.
Resolving a sound involves an unavoidable trade-off: the finer the resolution in time, the coarser it becomes in frequency, and the other way round. The real question is not where to compromise, but why only one can be chosen.
Sharp in Time, Blurred in Frequency
Listen to only a brief instant and you can say exactly when the click occurred. But in so short a stretch the pitch cannot be made out at all: overtones crowd together and there is no telling which band is resonating.
Sharp in Frequency, Blurred in Time
Listen across a long stretch and pitch and resonance separate finely, but a brief impact is flattened out: when it struck, how many times, how fast it decayed — all lost, and those are exactly what reveal a loose assembly.
Our approach: several scales at once, and the model learns which to trust
Several Scales at Once
Rather than choosing, the same sound is resolved at once on several spans. The longer spans hold pitch and resonance steady; the shorter ones pin down the moment of impact and its decay, and both stay available.
The Model Learns in Training Which Scale to Trust
At the instant of an impact the shorter spans should be trusted; for a sustained squeal, the longer ones. This is not a hand-tuned rule: the model learns it for itself from normal sound during training, so nothing has to be adjusted station by station.
How the Same Sound Changes Across Spans
The model also attends to how the same sound changes as the span shortens. An impact stands out more as the span shortens; a sustained sound stays steady across every span. That pattern of change is itself the most direct clue, and it too is learned into the representation.
Self-Supervised Pre-Training
Core ability comes from self-supervised pre-training, not from labelled defects.
The first question customers ask is how many defective parts to supply. At the core of the engine is a deep neural network, pre-trained self-supervised on a large body of unlabelled industrial sound to learn what normal sounds like, without relying on defect labels.
The Self-Supervised Pre-Training Loop
-
01
Listen
A segment of line sound goes in, and the network forms its own representation of it.
-
02
Reconstruct
It must then redraw the original from that representation alone, without hearing it again.
-
03
Compare
The reconstruction is compared with the original; the model can tell where the gaps are.
-
04
Correct
The mismatches become the direction for improvement, and the cycle starts again.
Why This Teaches Something Useful
Learned Beats Hand-Designed
In the traditional approach a person decides which indicators to rely on, so any regularity nobody thought of never enters the criteria. To redraw a machine sound faithfully, the network must learn harmonic families, envelope shape, resonant lift and noise-floor texture for itself — including cues no one thought to specify.
Know Normal Before You Notice Abnormal
An abnormal noise stands out because it departs from what it should have been. Once normal is learned, that departure has a reference, even for an unseen defect.
An Analogy
Humming back a melody you just heard shows you grasped its structure, not merely its title. Someone who only memorises titles is helpless with an unfamiliar tune.
What This Means on the Line
Getting Started Does Not Hinge on Defect Samples
Defective parts are rare, scattered across types, and rarely fully labelled. The introduction phase mainly needs normal sound representing current conditions.
Pre-Training First, On-Site Adaptation After
Pre-training learns a general representation across a wide range of industrial machine sound; on-site adaptation then fits it to each line, so it is not pulled off course by one line's quirks.
Label-Free Does Not Mean Context-Free
Pre-training provides the basic listening ability. Judging on a specific line still needs that line's normal sound as a reference and stable recording conditions.
Covering the Whole: Three Complementary Views
One inference, read three ways — a defect obvious in one view can be invisible in another.
All three views are read from the internal representation of one and the same deep neural network, not from three sets of hand-designed features. The same representation, read differently, tells apart different things — which is why they can share a single inference and still complement rather than repeat each other.
What Each View Picks Up
Overall Sound Signature
A wide shot of the segment. It is the steadiest path, and the first one to trust during a cold start.
Perceptually Aligned View
The same representation is rearranged along directions tied to hearing, then compared — like swapping in a ruler with a better-suited scale.
Temporal Texture
A summary smooths movement away; this path deliberately keeps the rhythm and any periodicity.
| View | Answers best | Intuition |
|---|---|---|
| Overall Signature | Drifted from normal as a whole? | A full-length photo: build is obvious, a small mark is not |
| Perceptually Aligned | Would a person hear it as harsh? | The same object, measured with a better-suited scale |
| Temporal Texture | Continuous, or in bursts? | Not a photograph, but the rhythm of a video |
Complementary, Not Redundant
Some defects barely register in the overall summary yet stand out plainly in temporal texture, and the reverse happens too. Since no one can predict which view will reveal a given defect, all three stay open.
Cost and Preconditions
One Shared Inference, Three Separate Scores
All three read the same inference, so only the readout step is added, with little effect on cycle time. They differ in scale and stability, and strung together the largest would dominate, so each is scored and calibrated on its own and combined at the score level.
Aligned With the Human Ear
Judging abnormal noise depends on roughness, fluctuation and sharpness, not on level.
On the line the final judge is a person, not a reading on a level meter. The engine has to align at the layer of how something sounds.
Two Sounds, Equally Loud
Two sounds can carry identical energy, one calm and one harsh. The difference lies in how that energy is spread, how it fluctuates, and whether it is modulated.
What the Ear Actually Cares About
These are publicly known perceptual attributes, used by the engine as reference directions.
How the Engine Uses These Attributes
A Reference Direction, So Magnification Is Targeted
The engine does not judge from these attributes directly; it uses them as a guide to pick out the hearing-related components inside the representation the deep neural network has learned and bring them onto a common scale. Magnify everything indiscriminately and the noise grows too, so the work is selection.
It Makes Existing Information Legible
Most of the difference between sounds on a line comes from background and conditions: temperature, lot variation, neighbouring stations. The defect's share is far smaller and is buried in a direct comparison, like weighing a feather on a heavy-duty scale. The scale is not broken; its graduations do not suit the task.
Aligning with perception amounts to picking up a better-suited ruler: the data is unchanged, yet masked differences return to a scale where they can be told apart.
The Ruler Must Be Fitted On Site
The graduations here are set by on-site adaptation from normal sound on the current line under current conditions. A change of line, station or conditions calls for re-setting them.
Judging Reliably: Score-Level Fusion and Per-Line Selection
Each viewpoint forms a complete judgment, the three meet at the score level, and the verdict lands on the part.
Once three viewpoints exist, the question becomes how to turn three opinions into one conclusion. Three experienced inspectors listening to the same part should not have their ears stitched together; each reaches a conclusion, and the opinions are pooled.
Do Not Concatenate
The three viewpoints differ in scale, so concatenating them lets the widest-ranging one dominate while the subtle differences of the other two flatten into noise. It is like summing body temperature, blood pressure, and a hearing test: each must first be converted into how far it deviates from a healthy population.
Concatenate the Representations
The dominant viewpoint decides everything, the quieter ones are drowned out, and trouble is hard to trace back to a single viewpoint.
Not Used
Score Separately, Then Combine
Each viewpoint answers on its own how far the part sits from normal on this line, and the answers are converted into comparable positions before being combined at the score level, so every viewpoint keeps its resolving power.
Our Approach
Per-Line Selection
The dominant defect type differs by product and by station: on one line it is a shift in overall timbre, on another it is fine fluctuation over time. So the engine provides one common mechanism and lets each line use its own evidence to choose the viewpoints it relies on. The scores come from model inference, not from hand-set thresholds; the only knob left to people is the false-call budget.
During recording a part is divided into several clips scored separately, and if any one clip is abnormal the part is abnormal. A fault often appears for only an instant and must not be averaged away.
Keeping Pace: Rolling Calibration and the False-Alarm Budget
The threshold is re-derived from recent good parts, and the quality-throughput trade-off is set by the customer with one dial.
A production line is a living thing: lots, seasons, and fixture wear all shift the baseline of the sound. A threshold fixed at the factory will sooner or later be too loose or too tight.
The Threshold Comes from Recent Good Parts
The engine gathers the scores of the line's recent good parts into a distribution of normal, and the threshold sits on it and is re-derived each day. Slow drift from lots and seasons is absorbed naturally. Rather than measuring every year's production with a ruler fixed at the factory, the line measures today's good parts each morning and then decides today's boundary. Calibration re-derives only the threshold; the pre-trained deep listening model and its model version stay unchanged.
The False-Alarm Budget: One Dial
Missed defects and false alarms cannot both be minimized; that is a trade-off, not a flaw. You set what share of good parts you accept being held for re-inspection, and the day's threshold is derived from it. Tighten it and workload rises; loosen it and throughput eases while borderline faults are likelier to pass.
Adjustable
Different stations and phases can be set to different strictness. Pilot and mass production need not share one setting.
Explainable
The threshold corresponds to re-inspection workload, to parts a person confirms one by one, and can be discussed directly in a quality review.
Reviewable
The gap between the setting and the share actually held is a health signal. If it keeps widening, check recording conditions and process changes before touching the threshold.
Review Results Flow Back: Closing the Loop
Held parts are re-inspected by people on the floor, and their conclusions flow back as confirmed samples. The system thus comes to learn what a real fault on this line sounds like.
What flows back is mainly the parts the system held, so the picture of abnormality it learns depends on the scope and quality of re-inspection. A fault never held rarely appears in the learning material at all. In the early phase, sample passed parts as well and feed those conclusions back too.
Running Stably: Site Prerequisites and Scope of Application
Stable operation depends half on the algorithm and half on whether recording and operating conditions hold steady.
Every acoustic inspection rests on one premise: what is heard this time is comparable with what was heard last time. Stating the limits is more useful than claiming there are none.
| Prerequisite | Why It Matters | What to Do on Site |
|---|---|---|
| Consistent Recording Setup | A change of microphone, placement, orientation, gain, or fixture shifts both absolute level and timbre. | Treat it as a new deployment: recalibrate, rebuild the baseline, and record what changed and when. |
| Stable Noise and Operating Conditions | Ordinary background fluctuation can be absorbed; a change of cycle time or a new noise source cannot. | Keep cycle and background steady; revalidate whenever a clear change occurs. |
| Clips Aligned to the Process Cycle | Clips too short fragment the evidence; clips too long average away momentary faults. | Agree the recording trigger and the clip boundaries with process engineering during deployment. |
| Baseline Accumulation and On-Site Adaptation | On-site adaptation uses good parts to align the pre-trained model to this line; while the baseline is thin, chance variation pulls decisions off course. | Accumulate first, cover the different shifts, complete on-site adaptation, then go live. The model version is fixed and inference is reproducible under the same version; a new line is brought up by on-site adaptation rather than by rewriting rules. |
| Shared Criteria and Review | If upstream and downstream disagree on what counts as abnormal, the samples fed back contradict one another. | Define the criteria and the responsible role, and have every shift use the same standard. |
Deployment: The Shadow Period
Run in Parallel, Record Only
The engine runs alongside the existing process; its output is recorded only and does not affect release.
Accumulate and Compare
The baseline takes shape and on-site adaptation completes, while decisions are compared part by part with the site's own to confirm shared criteria.
Switch Into Live Judgment
After the customer approves, it judges under the agreed false-alarm budget, and can return to record-only at any time.
The real value of the shadow period is aligning the system and the site on what counts as abnormal.
Scope and Limits
If any prerequisite above changes, recalibrate and run a shadow review before returning to live judgment, so every decision keeps resting on comparable audio.
Frequently Asked Questions
The questions we are asked most often before deployment, answered as things stand, with no promises of performance.
These questions come from real concerns on the floor. Actual behavior should be verified on your own line, through a shadow period.
Conventional methods have people decide in advance which indicators to rely on and set a threshold on each, so an anomaly nobody anticipated never enters the criteria. This engine is a deep learning system: pre-trained on large amounts of unlabelled sound, it learns for itself what to listen for, its representation, and is then adapted on site to the line. That is how it tells apart differences such as timbre changing while energy does not.
The main learning is self-supervised: it first learns what normal sound looks like, so it can start without labeled defects. A few confirmed abnormal parts help align criteria, but are not a precondition.
The threshold is re-derived continuously from recent good parts, so slow change is absorbed naturally. A substantive change to process, fixture, or recording setup calls for recalibration and a shadow review.
Every inference keeps its recording, clip, and score for review, and a wrong call feeds back into calibration; the final say remains with the site.
Inference runs after recording finishes, with the goal of completing within the station's cycle; edge or server is decided by project conditions.
You can trace which clip triggered the flag and which viewpoint deviated most, and play back that audio. This points to evidence rather than cause; the cause is still judged on the floor.
The sequence is fixed: connect recording, accumulate the baseline, validate in shadow, then switch into live judgment. How long it takes depends on output and how often faults occur.
Results are linked by part identifier, and the decision and score are output for the host system to record and aggregate. Interface format and trigger are agreed within the project.
The underlying deep neural network stays stable; an update only adds new ways of reading it out. A new model version can be shadow-validated in parallel, switched over after agreement, and rolled back if needed.
For feasibility on a specific line, we suggest a shadow period as the shared basis; a conclusion from your own products beats any general description.