IMAP TODOs #36

New Issue

Manos Pitsidianakis · 2019-12-07T22:00:34+02:00

2019-12-07 22:00:34 +02:00

~~IMAP LIST Extension for Special-Use Mailboxes~~ https://tools.ietf.org/html/rfc6154
* ~~Special use configuration should move entirely to melib and be modifiable from within the backend. Then IMAP and JMAP can use their special role metadata to update the folder confs~~ 2b6f6ab42c
Creation/deletion modification for mailboxes (requires melib support first)
~~Tagging~~ 0eaf178
* needs to parse custom flags as keywords in imap/protocol_parser.rs
~~password command in configuration~~ 6ca8c3b

- [x] **~~IMAP LIST Extension for Special-Use Mailboxes~~** https://tools.ietf.org/html/rfc6154 * ~~Special use configuration should move entirely to `melib` and be modifiable from within the backend. Then IMAP and JMAP can use their special role metadata to update the folder confs~~ 2b6f6ab42c4278cc2ff5f8c1819979d37b22ec50 - [x] Creation/deletion modification for mailboxes (requires `melib` support first) - [x] ~~Tagging~~ 0eaf178 * needs to parse custom flags as keywords in `imap/protocol_parser.rs` - [x] ~~password command in configuration~~ 6ca8c3b

Manos Pitsidianakis added the

help wanted

label 2019-12-07 22:00:34 +02:00

Manos Pitsidianakis referenced this issue from a commit

2019-12-18 08:59:21 +02:00

melib: Add BackendFolder methods, move special usage logic to backend - add count() method to return (unseen, total) counts - add is_subscribed() - add set_special_usage() and set_is_subscribed() concerns #8

oblitum commented

2020-03-07 05:24:45 +02:00

Please add support for password command in configuration to TODO list.

oblitum commented

2020-03-09 09:25:11 +02:00

Another feature request is to provide tree view of mailbox folders instead of flat view. To see for example Tagged/foo and Tagged/bar (or Tagged.foo and Tagged.bar if not using LAYOUT=fs) as a directory structure.

Another feature request is to provide tree view of mailbox folders instead of flat view. To see for example Tagged/foo and Tagged/bar (or Tagged.foo and Tagged.bar if not using `LAYOUT=fs`) as a directory structure.

Manos Pitsidianakis commented

2020-03-10 00:10:05 +02:00

Poster

Please add support for password command in configuration to TODO list.

Added!

Another feature request is to provide tree view of mailbox folders instead of flat view. To see for example Tagged/foo and Tagged/bar (or Tagged.foo and Tagged.bar if not using LAYOUT=fs) as a directory structure.

It should be a tree view and not flat. What separator does your account use? Can you provide a screenshot? An IMAP LIST could also be of help.

> Please add support for password command in configuration to TODO list. Added! > Another feature request is to provide tree view of mailbox folders instead of flat view. To see for example Tagged/foo and Tagged/bar (or Tagged.foo and Tagged.bar if not using `LAYOUT=fs`) as a directory structure. It should be a tree view and not flat. What separator does your account use? Can you provide a screenshot? An IMAP LIST could also be of help.

oblitum commented

2020-03-10 01:22:58 +02:00

Added!

Nice :)

It should be a tree view and not flat. What separator does your account use? Can you provide a screenshot? An IMAP LIST could also be of help.

Actually I'm not yet running meli, because I stopped my configuration when I noticed it required plain text password, so I'm waiting on that to actually migrate from aerc, which I've been using around a month or so, but meli seems more promising.

I'm using / as separator (with actual maildir fs layout), but I have no idea whether meli already should present it in directory structure, I just requested that because most programs I use, including K-9 and aerc, doesn't display it as a tree, but if meli already does it, then fine.

> Added! Nice :) > It should be a tree view and not flat. What separator does your account use? Can you provide a screenshot? An IMAP LIST could also be of help. Actually I'm not yet running meli, because I stopped my configuration when I noticed it required plain text password, so I'm waiting on that to actually migrate from aerc, which I've been using around a month or so, but meli seems more promising. I'm using `/` as separator (with actual maildir fs layout), but I have no idea whether meli already should present it in directory structure, I just requested that because most programs I use, including K-9 and aerc, doesn't display it as a tree, but if meli already does it, then fine.

Manos Pitsidianakis commented

2020-03-11 12:08:23 +02:00

Poster

Alright then.

As far as password commands are concerned, it's trivial to add but it has to be good UX. I'll test it with gpg and pinentry.

Alright then. As far as password commands are concerned, it's trivial to add but it has to be good UX. I'll test it with gpg and pinentry.

👍 1

Manos Pitsidianakis commented

2020-03-12 09:51:17 +02:00

Poster

6ca8c3b Added server_password_command for IMAP only. Tested with command:

gpg2 -r id@email.tld -dq /home/user/password.gpg

Trailing white space from the output is trimmed.

6ca8c3b Added `server_password_command` for IMAP only. Tested with command: ```sh gpg2 -r id@email.tld -dq /home/user/password.gpg ``` Trailing white space from the output is trimmed.

oblitum commented

2020-03-12 18:10:07 +02:00

Okay, I just tried and have hit a few issues. First is that I can't set a command like "pass foo | head -1", piping is not supported (it works in aerc, which specifies the string to be passed to sh -c for execution), so I had to save it to a script and set /my/script as the password command.

Once logged in, I only got a handful of folders, just Draft, Junk, Sent and Trash, I didn't get INBOX nor any of the other Tagged/ sub-folders I have. My dovecot maildir format is set like this: mail_location = maildir:/var/mail/%d/%n:LAYOUT=fs:INBOX=/var/mail/%d/%n/INBOX, which means I'm using filesystem layout, and INBOX is in its own folder.

I think mailer_cmd should be possible to be configured specific of accounts. I can't have a single mailer_cmd when using many accounts because each requires a different command for setting up password, etc, for msmtp.

Configuration:

[accounts."myaccount"]
root_mailbox = "INBOX"
format = "imap"
server_hostname="myserver"
server_password_command="mycmd"
server_username="myuser"
server_port="993" # imaps
index_style = "Conversations"
identity = "myident"
display_name = "myname"
subscribed_mailboxes = ["*"]

[composing]
mailer_cmd = 'mysendmail'

Okay, I just tried and have hit a few issues. First is that I can't set a command like `"pass foo | head -1"`, piping is not supported (it works in `aerc`, which specifies the string to be passed to `sh -c` for execution), so I had to save it to a script and set `/my/script` as the password command. Once logged in, I only got a handful of folders, just `Draft`, `Junk`, `Sent` and `Trash`, I didn't get `INBOX` nor any of the other `Tagged/` sub-folders I have. My dovecot maildir format is set like this: `mail_location = maildir:/var/mail/%d/%n:LAYOUT=fs:INBOX=/var/mail/%d/%n/INBOX`, which means I'm using filesystem layout, and `INBOX` is in its own folder. I think `mailer_cmd` should be possible to be configured specific of accounts. I can't have a single `mailer_cmd` when using many accounts because each requires a different command for setting up password, etc, for `msmtp`. Configuration: ``` [accounts."myaccount"] root_mailbox = "INBOX" format = "imap" server_hostname="myserver" server_password_command="mycmd" server_username="myuser" server_port="993" # imaps index_style = "Conversations" identity = "myident" display_name = "myname" subscribed_mailboxes = ["*"] [composing] mailer_cmd = 'mysendmail' ```

Manos Pitsidianakis commented

2020-03-13 15:52:25 +02:00

Poster

Thank you so much for your report,

piping is not supported

Yeah sh -c "$CMD" should be the solution, though the arguments are split by whitespace right now so my syntax would have issues.

Once logged in, I only got a handful of folders, just Draft, Junk, Sent and Trash, I didn't get INBOX nor any of the other Tagged/ sub-folders I have. My dovecot maildir format is set like this: mail_location = maildir:/var/mail/%d/%n:LAYOUT=fs:INBOX=/var/mail/%d/%n/INBOX, which means I'm using filesystem layout, and INBOX is in its own folder.

Could you post a censored IMAP LIST response, if possible?

The LIST response would exist in the debug logs, enabled by the debug-tracing compilation flag. Running cargo run --features=debug-tracing 2> debug.log should be enough
Alternatively, use openssl like openssl s_client -crlf -connect imap.server.tld:993 to connect (instructions)

I think mailer_cmd should be possible to be configured specific of accounts. I can't have a single mailer_cmd when using many accounts because each requires a different command for setting up password, etc, for msmtp.

It should be, yes. In general you can override settings on a folder basis, and you could override mailer_cmd for each account. This must be fixed!

Thank you so much for your report, > piping is not supported Yeah `sh -c "$CMD"` should be the solution, though the arguments are split by whitespace right now so my syntax would have issues. > Once logged in, I only got a handful of folders, just `Draft`, `Junk`, `Sent` and `Trash`, I didn't get `INBOX` nor any of the other `Tagged/` sub-folders I have. My dovecot maildir format is set like this: `mail_location = maildir:/var/mail/%d/%n:LAYOUT=fs:INBOX=/var/mail/%d/%n/INBOX`, which means I'm using filesystem layout, and `INBOX` is in its own folder. Could you post a censored IMAP `LIST` response, if possible? * The `LIST` response would exist in the debug logs, enabled by the `debug-tracing` compilation flag. Running `cargo run --features=debug-tracing 2> debug.log` should be enough * Alternatively, use `openssl` like `openssl s_client -crlf -connect imap.server.tld:993` to connect ([instructions](https://tewarid.github.io/2011/05/10/access-imap-server-from-the-command-line-using-openssl.html)) > I think `mailer_cmd` should be possible to be configured specific of accounts. I can't have a single `mailer_cmd` when using many accounts because each requires a different command for setting up password, etc, for `msmtp`. It should be, yes. In general you can override settings on a folder basis, and you could override `mailer_cmd` for each account. This must be fixed!

oblitum commented

2020-03-13 18:25:54 +02:00

Logged response is this:

["main"] melib/src/backends/imap/connection.rs:362_17:	sent: M3 LIST "" "*"
["main"] melib/src/backends/imap/connection.rs:317_33:	&ret[last_line_idx..] = "M3 OK List completed (0.003 + 0.574 + 0.576 secs).\r\n"
["main"] melib/src/backends/imap/protocol_parser.rs:115_9:	&val = "M3 OK List completed (0.003 + 0.574 + 0.576 secs).\r\n"
["main"] melib/src/backends/imap.rs:793_9:	out: * LIST (\HasNoChildren \Junk) "/" Junk
* LIST (\HasNoChildren \Sent) "/" Sent
* LIST (\HasNoChildren \Trash) "/" Trash
* LIST (\HasNoChildren \UnMarked \Drafts) "/" Drafts
* LIST (\Noselect \HasChildren) "/" Tagged
* LIST (\HasNoChildren) "/" Tagged/subbox1
* LIST (\HasNoChildren) "/" Tagged/subbox2
* LIST (\HasNoChildren) "/" Tagged/subbox3
* LIST (\HasNoChildren) "/" Tagged/subbox4
* LIST (\HasNoChildren) "/" INBOX
M3 OK List completed (0.003 + 0.574 + 0.576 secs).

This is the full log redacted:

Single time download will expire in 24h

Logged response is this: ``` ["main"] melib/src/backends/imap/connection.rs:362_17: sent: M3 LIST "" "*" ["main"] melib/src/backends/imap/connection.rs:317_33: &ret[last_line_idx..] = "M3 OK List completed (0.003 + 0.574 + 0.576 secs).\r\n" ["main"] melib/src/backends/imap/protocol_parser.rs:115_9: &val = "M3 OK List completed (0.003 + 0.574 + 0.576 secs).\r\n" ["main"] melib/src/backends/imap.rs:793_9: out: * LIST (\HasNoChildren \Junk) "/" Junk * LIST (\HasNoChildren \Sent) "/" Sent * LIST (\HasNoChildren \Trash) "/" Trash * LIST (\HasNoChildren \UnMarked \Drafts) "/" Drafts * LIST (\Noselect \HasChildren) "/" Tagged * LIST (\HasNoChildren) "/" Tagged/subbox1 * LIST (\HasNoChildren) "/" Tagged/subbox2 * LIST (\HasNoChildren) "/" Tagged/subbox3 * LIST (\HasNoChildren) "/" Tagged/subbox4 * LIST (\HasNoChildren) "/" INBOX M3 OK List completed (0.003 + 0.574 + 0.576 secs). ``` This is the full log redacted: - [Single time download will expire in 24h](https://send.firefox.com/download/4b6be77e9fbdf042/#CyHOewVnALq9DREhwdX4hQ)

oblitum commented

2020-03-13 18:29:08 +02:00

It's noticeable that the folders that are displayed have this \Foo) "/" Foo in the LIST response. I'm not sure whether at least INBOX too should also have come with that or not, but it only makes a difference in meli.

It's noticeable that the folders that are displayed have this `\Foo) "/" Foo` in the LIST response. I'm not sure whether at least INBOX too should also have come with that or not, but it only makes a difference in `meli`.

Manos Pitsidianakis commented

2020-03-14 10:57:39 +02:00

Poster

@oblitum It seems the problem is you are not subscribed to those mailboxes. Look for the LSUB response in your debug logs. I'm not sure how to deal with this since it's not completely wrong to assume you're not interested in unsubscribed mailboxes, yet it's not correct either. Perhaps there could be an entry like "... and n others" in the sidebar. And a more complete list of mailboxes in the account status.

In any case, you can alter subscription status with {un,}subscribe-mailbox ACCOUNT_NAME MAILBOX_PATH

PS: Btw, the attributes you're referring to is RFC 6154 SPECIAL-USE which is supported.

@oblitum It seems the problem is you are not subscribed to those mailboxes. Look for the `LSUB` response in your debug logs. I'm not sure how to deal with this since it's not completely wrong to assume you're not interested in unsubscribed mailboxes, yet it's not correct either. Perhaps there could be an entry like "... and `n` others" in the sidebar. And a more complete list of mailboxes in the account status. In any case, you can alter subscription status with `{un,}subscribe-mailbox ACCOUNT_NAME MAILBOX_PATH` PS: Btw, the attributes you're referring to is [RFC 6154 SPECIAL-USE](https://tools.ietf.org/html/rfc6154) which is supported.

oblitum commented

2020-03-14 18:38:30 +02:00

But I have subscribed_mailboxes = ["*"] in my configuration.

But I have `subscribed_mailboxes = ["*"]` in my configuration.

oblitum commented

2020-03-14 18:43:11 +02:00

I've set it in specific form subscribed_mailboxes = ["INBOX", "Drafts", "Tagged/meli"], and still only Drafts show up.

I've set it in specific form `subscribed_mailboxes = ["INBOX", "Drafts", "Tagged/meli"]`, and still only `Drafts` show up.

oblitum commented

2020-03-14 18:48:26 +02:00

And when executing the subscribe-mailbox ACCOUNT_NAME INBOX command from TUI, after confirmation box, I get Not Implemented as result.

And when executing the `subscribe-mailbox ACCOUNT_NAME INBOX` command from TUI, after confirmation box, I get `Not Implemented` as result.

oblitum commented

2020-03-16 19:34:48 +02:00

I got INBOX listed!

Looks like the way I've setup dovecot with standalone INBOX folder, it doesn't get subscribed by default. After doveadm mailbox subscribe -u me@domain.tld INBOX, meli starts listing it.

I'm a bit confused, even though it was unsubscribed, why didn't it change to subscribed since I've configured it as subscribed on meli config? From what I can tell this looks like two levels of subscription access, one on the server, with higher priority, and another on the client.

Another interesting thing. To get subfolders listed, I had to subscribe to the parent folder. This is why any of the Tagged/boxN were not being listed. I had to doveadm mailbox subscribe -u me@domain.tld Tagged and then doveadm mailbox subscribe -u me@domain.tld Tagged/box1, afterwards, I need to subscribed_mailboxes = ["*"] or subscribed_mailboxes = ["INBOX", "Tagged", "Tagged/box1"] (parent folder also needs to be listed here).

If I simply remove subscribed_mailboxes, then I get folder listing as configured with doveadm.

All in all, now that I see how to configure it, I'm able to play with meli :)

FWIW, the tree view of subfolders is working as expected.

I got `INBOX` listed! Looks like the way I've setup dovecot with standalone `INBOX` folder, it doesn't get subscribed by default. After `doveadm mailbox subscribe -u me@domain.tld INBOX`, `meli` starts listing it. I'm a bit confused, even though it was unsubscribed, why didn't it change to subscribed since I've configured it as subscribed on `meli` config? From what I can tell this looks like two levels of subscription access, one on the server, with higher priority, and another on the client. Another interesting thing. To get subfolders listed, I had to subscribe to the parent folder. This is why any of the `Tagged/boxN` were not being listed. I had to `doveadm mailbox subscribe -u me@domain.tld Tagged` and then `doveadm mailbox subscribe -u me@domain.tld Tagged/box1`, afterwards, I need to `subscribed_mailboxes = ["*"]` or `subscribed_mailboxes = ["INBOX", "Tagged", "Tagged/box1"]` (parent folder also needs to be listed here). If I simply remove `subscribed_mailboxes`, then I get folder listing as configured with `doveadm`. All in all, now that I see how to configure it, I'm able to play with meli :) FWIW, the tree view of subfolders is working as expected.

oblitum commented

2020-03-16 19:59:48 +02:00

Actually I'm waiting on per-account mailer_cmd for it to be usable daily. It would be nice to have more Vim-like bindings option too, like i for command instead of space; j, k instead of arrows, etc, but I guess I can configure those myself.

Actually I'm waiting on per-account `mailer_cmd` for it to be usable daily. It would be nice to have more Vim-like bindings option too, like `i` for command instead of `space`; `j`, `k` instead of arrows, etc, but I guess I can configure those myself.

Manos Pitsidianakis commented

2020-03-17 16:39:20 +02:00

Poster

I'm a bit confused, even though it was unsubscribed, why didn't it change to subscribed since I've configured it as subscribed on meli config? From what I can tell this looks like two levels of subscription access, one on the server, with higher priority, and another on the client.

That's true. Maildir for example doesn't have any notion of mailbox subscription whereas IMAP and JMAP does. It's confusing, the terminology must change.

Another interesting thing. To get subfolders listed, I had to subscribe to the parent folder

That's also a confusing thing: what to do in this case. The unsubscribed ancestors could be flattened to a dummy unselectable mailbox.

Perhaps making meli-level subscription opt out instead of opt in would be more intuitive.

Actually I'm waiting on per-account mailer_cmd for it to be usable daily

I am working on a configuration revision right now, to make overriding better.

> I'm a bit confused, even though it was unsubscribed, why didn't it change to subscribed since I've configured it as subscribed on meli config? From what I can tell this looks like two levels of subscription access, one on the server, with higher priority, and another on the client. That's true. Maildir for example doesn't have any notion of mailbox subscription whereas IMAP and JMAP does. It's confusing, the terminology must change. > Another interesting thing. To get subfolders listed, I had to subscribe to the parent folder That's also a confusing thing: what to do in this case. The unsubscribed ancestors could be flattened to a dummy unselectable mailbox. Perhaps making meli-level subscription opt out instead of opt in would be more intuitive. > Actually I'm waiting on per-account mailer_cmd for it to be usable daily I am working on a configuration revision right now, to make overriding better.

Manos Pitsidianakis added this to the alpha milestone 2020-03-17 17:31:48 +02:00

Manos Pitsidianakis commented

2020-03-18 19:39:08 +02:00

Poster

@oblitum 9ff54f236b introduces composing settings override per-account, so a setting like follows should work:

[accounts.accountA]
root_mailbox = "/home/epilys/mail/"
format = "maildir"
listing.index_style = "compact"
composing.mailer_cmd = '/bin/true'
...

[composing]
mailer_cmd = 'msmtp --read-recipients --read-envelope-from'

@oblitum 9ff54f236b59e4bb9684353121e5686da08e26a5 introduces composing settings override per-account, so a setting like follows should work: ``` [accounts.accountA] root_mailbox = "/home/epilys/mail/" format = "maildir" listing.index_style = "compact" composing.mailer_cmd = '/bin/true' ... [composing] mailer_cmd = 'msmtp --read-recipients --read-envelope-from' ```

oblitum commented

2020-03-19 02:04:19 +02:00

@epilys, thx. It's still a little difficult to adopt it at this phase :/ very experimental. I'm having issues with syncing and another thing is that it takes a lot of processing when starting, and keep doing that after couple of minutes.

Manos Pitsidianakis commented

2020-03-19 12:32:08 +02:00

Poster

@oblitum testing still needs to be done, yes.

I'm having issues with syncing and another thing is that it takes a lot of processing when starting

Are you using a debug or release build?

There's still header caching to be done properly, and IMAP needing to re-download everything is a bottleneck. Clients usually fake stuff by showing entries that are not loaded yet; I haven't done anything in that direction yet.

@oblitum testing still needs to be done, yes. >I'm having issues with syncing and another thing is that it takes a lot of processing when starting Are you using a debug or release build? There's still header caching to be done properly, and IMAP needing to re-download everything is a bottleneck. Clients usually fake stuff by showing entries that are not loaded yet; I haven't done anything in that direction yet.

Manos Pitsidianakis closed this issue

2020-07-05 22:00:42 +03:00

Sign in to join this conversation.