The Sensor Model Was Learning When We Sampled

The first clue did not come from a training dashboard but it came from field tests that did not make physical sense.

A model intended to distinguish broad real-world states produced inconsistent activity predictions. Stationary situations sometimes received unexpectedly strong scores for moving-vehicle categories. Phone-placement predictions did not always match what the tester was doing.

Our first questions were conventional. Were the outputs in the wrong order? Was a label mapping incorrect? Was the model simply expressing low confidence?

Those were reasonable possibilities. But the more consequential problem was hiding one layer earlier, in how the training data had been collected.

Most of the original sensor windows came from a decision flow: the application reached a point where it might take an action, invoked Context SDK and created a sample. By definition, those were moments in which the phone was already being used.

We thought we had trained on a collection of sensor-derived real-world moments. More precisely, we had trained on real-world moments conditional on the application asking a question. That distinction changed the work.

The trigger was part of the dataset

Every dataset has a collection mechanism. In mobile AI, that mechanism can be unusually influential.

If sensor data is collected primarily when an application requests a decision, the dataset will contain more moments associated with active phone use. Situations in which the phone is not in someone’s hand - sitting in a bag during a journey, for example - will be represented less often unless another collection path captures them.

The model does not know why those moments are missing. It only sees the distribution it has been given.

This is a form of selection bias, but it has a particularly important consequence for situational awareness. The missing examples are not random rows. They may be entire classes of physical situations.

A model can be technically correct on the distribution it was trained to reconstruct while still being poorly prepared for the distribution it encounters in the world.

The first lesson from our investigation was therefore not about architecture. It was about instrumentation:

Before asking whether a context model understands a moment, ask what had to happen inside the application for that moment to enter the training corpus.

Expanding the world the model could see

We rebuilt the dataset using sensor windows from two different collection paths.

The first contained the existing decision-time samples. The second added samples from a background-triggered path, including more situations in which the phone was not actively in hand.

This was not merely a matter of adding volume. A large dataset can remain narrow if it repeatedly captures the same kind of moment. The objective was to increase situational coverage.

The investigation also exposed several more conventional - but still consequential - problems:

• Yaw needed to be represented relatively so that differences in initial phone orientation did not dominate the representation.
• Partial sensor windows required a different yaw calculation from the one used in the first retraining pass.
• Near-identical stationary-phone samples made up approximately 15% of the new merged dataset and were removed.
• Missing values represented by a sentinel value had affected the dataset statistics.
• Some samples had appeared in both the training and validation sets.

That last finding was especially important. The initial evaluation had made the improvement look larger than we could responsibly claim. So we fixed the split and reran the comparison.

The cleaned challenger still performed better than the previous model across the mixed dataset and across the two collection paths evaluated separately. But the leak changed how we treated the earlier numbers. They were useful for directing the investigation, not suitable as final evidence.

This is an uncomfortable distinction, but an essential one. A better-looking metric is not progress if the comparison is not trustworthy.

Changing the data before making the model more elaborate

The revised training also introduced weight decay and dropout, but the most important change was not a more exotic neural-network design. It was giving the model a more representative and less repetitive view of the world.

That matters for an on-device system.

A compact model cannot recover situations that are absent from its training corpus. More parameters might help it fit the available distribution, but they cannot create coverage of moments it has never seen.

The same principle applies to the learned representation. ContextSDK compresses short windows of sensor data into moment embeddings. Different activity heads can then use those embeddings to estimate aspects of the situation, such as how the phone is moving, its placement or the person’s broad body state.

If the moment embedding is biased toward active-use situations, every downstream head inherits that limitation.

Conversely, improving the representation can improve multiple heads without redesigning each one independently.

We saw an encouraging early indication of this effect. In a separate preliminary evaluation, replacing the earlier embedder improved the internal scores of all three activity heads tested:

These are directional internal results, not yet a publication-grade benchmark. The candidate embedder was still under development, and transport continued to struggle with physically similar categories such as trains and trams. The comparison needs to be repeated against the final cleaned model with the evaluation metric, class balance and data split fully specified.

But the pattern is worth investigating: better representations of the moment appeared to make several different interpretations of that moment easier.

Situational awareness begins before inference

It is tempting to describe situational awareness as an inference problem: sensor data enters, a model runs and a context prediction comes out.

Our experience suggests that the harder problem begins earlier.

Which moments are collected? Which are repeated thousands of times? Which situations exist only outside the application’s normal decision flow? Which device orientations are overrepresented? Does the validation set contain genuinely unseen moments, or merely another copy of something the model already encountered?

These choices determine what an on-device model has an opportunity to learn.

They also affect the decisions built on top of it. A timing system may rank whether an application should act, wait or get out of the way. But that decision score is only as useful as the representation underneath it. A timing layer cannot compensate for a physical situation that the model has rarely or never observed.

This does not imply that one universal context model can discover the right behavior for every application. Context remains app-specific. A moment that raises the value of an intervention in one product may lower it in another.

The shared requirement is more basic: the situational-awareness layer needs a sufficiently broad and honest representation of the moment before an application can learn what that moment means for its own decisions.

The sampling policy is part of the model

The field observations that began this investigation were not proof of a single cause. They were signals that the system’s picture of the world was incomplete.

Following those signals led us through the collection paths, the dataset, the sensor transformations and the evaluation setup. Along the way, we found both an obvious defect - a training/validation leak - and a less obvious design problem: the application trigger that created the data had shaped the situations represented in it.

The resulting model looks more promising. More importantly, the evaluation is now more credible and the remaining limitations are easier to name.

For teams building contextual intelligence on mobile devices, the practical question is not only:

How good is the model?

It is also:

What had to happen before the model was allowed to see the momen