Clarification on Definition of "On-Device Models" Based on TAG Feedback

[anssiko]

Opening this issue in response to the message from @toreini, queted below:

Hi folks,

I am reaching out to the group to seek clarification on the definition of "on-device models" in light of the recent feedback and discussion from the TAG regarding the place where models are run. Also, the TAG review on the API mentions preference for on-device models (as "We're concerned about the use of the network", may need you press the "load more" button to be able to see the exact TAG review).

Reviewing TAG discussions, there are several scenarios that could be considered as "on-device" processing, which include:

1. The same device that the UA (User Agent) is running on.
2. Encrypted connections to other devices that the same user owns, for example, to allow a phone to offload work to the same person's desktop.
3. Various flavours of private computation on other people's computers, ranging from homomorphic encryption to TEEs (Trusted Execution Environments).
4. "Service providers" (in the GDPR sense) for the user, who have committed not to use or expose the data outside that particular user's context.
5. Distinct data controllers (again in the GDPR sense), who could use the user's data for their own purposes.

Given these scenarios, I would like to understand how the group interprets and applies the term "on-device models." Specifically:

Scope of "On-Device": Does the group consider only scenario A (processing on the same device) as "on-device," or does it extend to scenarios B, C, and D as well?
Implications for Privacy and Security: How do these different scenarios impact privacy and security considerations for the models (we can use the writing assistant API models as usecase, but it can be extended)?
Alignment with TAG Recommendations: How does the group plan to align its definition and implementation of "on-device models" with the TAG's feedback and recommendations?

Clarifying these points can be stepping stone for further discussions on security and privacy aspects of on-device processing.

Best Regards,
Ehsan

Discussion and feedback welcome on these points.

[reillyeon]

This also came up in webmachinelearning/prompt-api#172 which re-enforces the need for a taxonomy. While it is possible to implement many of these APIs using either local or remote compute, the security and privacy considerations of these two implementation options differ greatly. I think we can only have a useful conversation if we consider them separately.

[jasonmayes]

Historically we have always defined as follows:

On-device AI (Client side Web AI): Using the same GPU/NPU/CPU on the client machine that is viewing the website (be that a mobile, tablet, or laptop etc). The inference can occur fully offline once the model is downloaded.

On-network AI: A term I coined when you need some middle path. This could be in the form of a localhost connection on the same device, or private LAN connectivity for mesh situations where cloud is not allowed - but trusted private networks are. This is useful in robotics where GPU on the robot may drain battery so you want to delegate compute elsewhere, but that low latency "brain" could live on a laptop or such that is wirelessly connected to the robot on a private network, but still not able to access the wider internet (cloud based providers).

Edge AI: Historically associated with lower powered IoT devices like a Raspberry Pi or such. These days I have noticed the term is expanding to be the umbrella term for this and all of the above.

Cloud AI: Requires a network connection to the public cloud services / wider internet. Must be online to use.