MBOX-Line: From asuth at mozilla.com  Tue Sep  3 03:53:00 2013
To: imap-protocol@u.washington.edu
From: Andrew Sutherland <asuth@mozilla.com>
Date: Fri Jun  8 12:34:51 2018
Subject: [Imap-protocol] Best way to support HTTP PUT based 'push'
	notifications from IMAP?
Message-ID: <5225BF8C.7060602@mozilla.com>

On devices that either have difficulty maintaining persistent TCP 
connections or have power concerns about doing so, maintaining an IMAP 
connection for IDLE/NOTIFY purposes is undesirable and it would be great 
if the IMAP server could generate a notification via other means.

For the Firefox OS e-mail client, one of the Web APIs the Firefox OS 
platform makes available and that Mozilla and other contributors are 
trying to standardize is the "Push API" (http://www.w3.org/TR/push-api/) 
under the auspices of the webapps WG (http://www.w3.org/2008/webapps/).  
The API is somewhat decoupled from the actual notification mechanism.  
Mozilla's reference server protocol is documented at 
https://wiki.mozilla.org/WebAPI/SimplePush/Protocol.

 From the IMAP server's perspective, the process is that the client uses 
the Push API to get a URL and then the client tells the IMAP server the 
URL.  To notify the client, the IMAP server issues a PUT request to the 
URL.  The only data payload is the 'version' which is a monotonically 
increasing 64-bit integer and allows for a degree of idempotency.  The 
only key thing for the version is that if the server doesn't increase 
the value since the last time the client heard it, the client won't 
receive a new notification.

Our application-level goal is that rather than checking every N minutes, 
after each synchronization attempt we refresh our notification request 
before disconnecting from the IMAP server.  The IMAP server would then 
send a ping notification when new messages are received for the 
account/inbox/whatever.

Details/thoughts from my investigation are found below.  In general, I'm 
very interested in what server implementers think and would be willing 
to adopt, especially for things that might be enabled-by-default in 
deployments.  It's been indicated to me that we (Mozilla) have 
engineering resources that could be used to help provide patches to open 
source implementation to help make this happen.

Note that any solution that satisfies our needs could probably also 
satisfy similar needs.  For example, the Firefox OS Calendar App makes 
use of the Google Calendar API's push notifications mechanism as 
documented at https://developers.google.com/google-apps/calendar/v3/push


# Existing Work

## XAPPLEPUSHSERVICE

The only thing I've seen in the wild seems to be XAPPLEPUSHSERVICE 
advertised by Yahoo's IMAP server and Apple's dovecot fork.  The most 
useful information I've found on this is 
http://www.dovecot.org/list/dovecot/2012-August/067682.html which leads 
us to the implementation of XAPPLEPUSHSERVICE at 
http://www.opensource.apple.com/source/dovecot/dovecot-239.8/dovecot/src/imap/cmd-x-apple-push-service.c 
which tells us the syntax of the command.  While the idea is similar, 
XAPPLEPUSHSERVICE does not seem to support alternate endpoints to 
Apple's Push Notification Service and so is not useful for our purposes.

## sieve/enotify

The sieve enotify extension (http://tools.ietf.org/html/rfc5435) seems 
like the only standadized mechanism capable of implementing such logic.  
Unfortunately, there is no HTTP/URL notification listed in the registry 
at 
https://www.iana.org/assignments/sieve-notification/sieve-notification.xhtml. 
'mailto' is a defined mechanism that has some support (ex: Pigeonhole on 
Dovecot), and it's possible for Mozilla to stand-up a proxy to help 
bridge this gap.

The main problem with sieve is that adoption does not seem particularly 
wide-spread and discoverability when supported is limited, although I'm 
hoping that http://tools.ietf.org/html/rfc6785 could help improve that.


# Potential Workarounds

I think the following options exist to make this work today:

## sieve mailto bridge

Stand up an SMTP-receiving proxy so that sieve/enotify/mailto can be 
used to trigger instead of HTTP PUT.  Use sieve to set up a rule to 
generate mail notifications every time a message is received.  No 
message contents would be sent.  This might result in a large number of 
notifications since it's not obvious to me how to write a stateful sieve 
script.

## Hosted IMAP Proxy (Undesirable due to inherent credential/data exposure)

A proposal from those working on the back-end of Mozilla's push 
notification service is an HTTPS-accessible proxy.  Instead of having 
the device maintain the long-lived IMAP connection, the device has a 
server do that on its behalf.  The very undesirable aspect of such a 
solution is that while the proxy need not persist the credentials to 
disk, it does need to know them to establish the connection using 
traditional LOGIN.  And even if a challenge-response authentication 
mechanism was used in such a way that the proxy server could not 
establish additional connections without device support, the proxy (or 
whatever pretends to be the proxy) would still have a fully authorized 
IMAP connection to use as it chooses until the connection is dropped.


# Potential Standards Approaches

## Create a new capability, ex: PUSH-NOTIFY-HTTP=PUT

Create a "PUSH-NOTIFY-HTTP PUT <URL> <BODY PAYLOAD>" capability with 
very simple semantics:
- Each request is one-shot.  The request lives until it is fired.
- At once most one request per URL; new requests replace the old request.
- An HTTP PUT is triggered to the given URL with content-type 
application/x-www-form-urlencoded with the provided body payload.
- Body payload is limited to something really small; we only need to be 
able to encode "version=<64bitnumber>"
- Delivery to the Inbox or maybe just delivery to the account in general 
in case a filter files the message somewhere.  Messages filed to the 
junk folder would not trigger the notification.

## sieve enotify HTTP PUT extension

Since it's not likely for adoption to happen immediately anyways and 
sieve is extremely powerful it seems like a reasonable choice to avoid 
introducing a new feature which is doomed to feature creep.

Because sieve is not particularly stateful the main issue would be how 
to achieve single-shot semantics.  This might be best addressed by doing 
something like the servermetadata example at 
http://tools.ietf.org/html/rfc5490#section-4.1 where the client would 
use metadata to control the process and the URL notification function 
would have built-in idempotent semantics.  Each time the client 
connected it would refresh the 'verson' it uses and which is used in the 
notification invocation.  Then when the sieve rule next fires, the 
URL/payload changes so that the notify action actually goes through.

Multiple consumers of such notifications would be handled during 
rewrites of the sieve rules.  A new device setting up for push 
notifications would identify its rule based on the URL the push API 
tells it.  It would write its rule to the sieve store along with the 
current timestamp for the rule.  It would also cull any other rules of 
the same variety whose timestamp more than twice as old as the 
rule-rewrite-interval based on the timestamp.

## custom sieve extension

Realize the above state management logic is too crazy.  Internalize the 
one-shot state management, still using metadata somehow.

## Extend NOTIFY somehow

This seems like it would be a mismatch since the NOTIFY subscription is 
inherently associated with the connection lifetime and any other 
lifetime would be weird.

## Define CONTEXT=pushnotify

Similar to NOTIFY, this seems like a mismatch.


Thanks for any/all responses,
Andrew Sutherland

