Calendaring and Email on Android with GCalendar and GMail

Its been a while since my last post, was busy completing the NLP and ML online Coursera courses. For what its worth, I loved the courses, and I think they are worth taking if you can make the necessary time/energy commitment (and if you want/need to know about this stuff, of course). The nice thing about both courses is that it provides you a broad (non industry specific) overview of the tools and techniques that are popular, and you can decide which ones you want to drill down on for your own needs. So its a bit easier than figuring out stuff from scratch on your own. But there is a time commitment required, and this is the first weekend in almost 4 months that I don't have homework to finish :-).

During this time, our phone contracts ran out, so my wife and I decided to upgrade from our Blackberries to Androids. Normally, I prefer to be waay behind the technology curve when it comes to stuff like this, but I had been wanting to get an Android for some time, and I figured its been around long enough for the obvious bugs to have been fixed. While that is true to some extent, I think that Android phones have still some ways to go compared to the Blackberry when it comes to the out-of-the-box experience with Email and Calendaring.

Its not as if my email/calendaring requirements are super special or anything. I have a personal account with Comcast that is served off a POP mail server and a work account served off an MS Exchange (IMAP) server. Android has a Mail app, which, like the Blackberry, results in two different Inboxes. Unlike the Blackberry, however, the Android Mail app has no support for Calendar Event reminders, which I have grown quite accustomed to.

Reading through various forums on the Internet, I found a setup, based on Google Calendar (GCalendar) and Google Mail (GMail) that was being used by some people who had similar requirements as mine. The diagram below shows the setup.

Mail Setup

GMail provides a way to pull in email from POP accounts, so I used that to pull down the emails from my personal mailbox on my ISP. Since GMail would be set up as an IMAP server, there was no need to keep email on the POP server after downloading.

GMail does not provide a way to pull emails from IMAP accounts such as Microsoft Exchange server that hosts my work email account. Based on the advice on the various forums I set up fetchmail on my work computer to fetch the work email at regular intervals and forward them to GMail. However, that has a side effect of marking all the fetched mail as read. Since the read/unread marker is a major navigation cue for me, this was unacceptable.

Unfortunately fetchmail does not offer a way to control this behavior, so I decided to write my own little Python script to pull emails at regular intervals from my Exchange server and forward to GMail. Before each run, the script reads the last UID (unique ID) retrieved by the previous invocation and retrieves mail that has a UID higher than that one. At the end of the run, it writes back the highest UID fetched into the file. Here is the script.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68

#!/usr/local/bin/python # Runs via cron every 10 minutes (crontab entry below) # */10 * * * * /home/spal/bin/copymail.py # import email import imaplib import smtplib import traceback # the file below contains the latest UID from previous run TS_FILE = "/home/spal/.android/copymail.ts" # configuration file, contains values for # LOCAL_IMAP_HOST = host name of Exchange IMAP server. # LOCAL_IMAP_USER = user name on Exchange IMAP server. # LOCAL_IMAP_PASS = password on Exchange IMAP server. # LOCAL_IMAP_INBOX = name of main Inbox directory on IMAP # LOCAL_IMAP_EMAIL = source email address of IMAP # SMTP_FORWARD_HOST = localhost (I had sendmail running) # TARGET_EMAIL_ADDRESS = target email to forward to CF_FILE = "/home/spal/.android/copymail.cf" def get_conf(): f = open(CF_FILE, 'r') conf = {} for line in f: if len(line.strip()) == 0: continue [key, value] = line[:-1].split("=") conf[key] = value f.close() return conf def read_uid(): f = open(TS_FILE, 'r') ts = f.read() f.close() return ts def write_uid(uid): f = open(TS_FILE, 'w') f.write(uid) f.close() def main(): conf = get_conf() mail = imaplib.IMAP4(conf["LOCAL_IMAP_HOST"]) mail.login(conf["LOCAL_IMAP_USER"], conf["LOCAL_IMAP_PASS"]) mail.select(conf["LOCAL_IMAP_INBOX"]) result, data = mail.uid("search", None, "ALL") avail_uids = data[0].split() uid = read_uid() smtp = smtplib.SMTP(conf["SMTP_FORWARD_HOST"]) for avail_uid in avail_uids: if int(avail_uid) > int(uid): result, data = mail.uid("fetch", avail_uid, "(RFC822)") raw_email = data[0][1] email_message = email.message_from_string(raw_email) from_name, from_addr = email.utils.parseaddr(email_message["From"]) try: smtp.sendmail(from_addr, [conf["TARGET_EMAIL_ADDRESS"]], \ raw_email.encode("ascii", "replace")) except smtplib.SMTPException: print "error: unable to send mail" print traceback.print_exc() write_uid(avail_uids[-1]) if __name__ == "__main__": main()

The script is run via cron every 10 minutes (crontab entry is available in the comments of the script above.

On the GMail side, we create a label "WORK" which is applied to the forwarded emails via a custom filter (which we also create). My custom filter sets the label to "WORK" when the To: address contains *@healthline.com. Not perfect, but I rarely get personal emails from work (colleagues who do email me on my personal account tend to use their personal accounts as well), so it works for me.

Since our GMail account uses IMAP, I can access my email from either my phone or notebook without worrying about email loss.

Calendar Setup

For the Calendaring, I initially tried using GCalDaemon, as advised on the forums, but I just could not get it to work. Apparently I had several unparseable (to GCalendar) Calendar events which caused GCalendar to hit its daily maximum limit. After some attempts, I just gave up and decided to write another script similar to the one above, that would periodically read my Evolution Calendar (.ics) file and push these events over to GCalendar via the GData API. Here is the script.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112

#!/usr/local/bin/python # Called by cron every 6 hours. Crontab entry: # 0 0,6,12,18 * * * /home/spal/bin/copycal.py # import atom import atom.service from datetime import date from datetime import datetime from datetime import time import gdata.calendar import gdata.calendar.service import gdata.service from icalendar import Calendar # Contains the timestamp from previous run. TS_FILE = "/home/spal/.android/copycal.ts" # Configuration parameters. # CALENDAR_FILE = /full/path/to/evolution/calendar.ics # GOOGLE_CALENDAR_EMAIL = GMail email # GOOGLE_CALENDAR_PASSWORD = Google Password # GOOGLE_CALENDAR_CLIENT_STR = some random client string # GOOGLE_CALENDAR_FEED_URL = /calendar/feeds/default/private/full CF_FILE = "/home/spal/.android/copycal.cf" def str_to_dttime(s): d = date(int(s[0:4]), int(s[4:6]), int(s[6:8])) t = time(int(s[9:11]), int(s[11:13]), int(s[13:15])) return datetime.combine(d, t) def dttime_to_str(dt): return dt.strftime("%Y%m%dT%H%m%S") def dttime_to_iso_zulu(dt): return dt.strftime("%Y-%m-%dT%H:%M:%S.000Z") def read_timestamp(): f = open(TS_FILE, 'r') ts = f.read() f.close() return ts def write_timestamp(ts): f = open(TS_FILE, 'w') f.write(dttime_to_str(ts)) f.close() def get_conf(): f = open(CF_FILE, 'r') conf = {} for line in f: if len(line.strip()) == 0: continue [key, value] = line[:-1].split("=") conf[key] = value f.close() return conf def main(): # read start timestamp from file and update with end timestamp # for the next run start_ds = str_to_dttime(read_timestamp()) end_ds = datetime.now() write_timestamp(end_ds) end_dttime = dttime_to_str(end_ds) # read configuration parameters conf = get_conf() # login to google calendar gcalendar = gdata.calendar.service.CalendarService() gcalendar.email = conf["GOOGLE_CALENDAR_EMAIL"] gcalendar.password = conf["GOOGLE_CALENDAR_PASSWORD"] gcalendar.source = conf["GOOGLE_CALENDAR_CLIENT_STR"] gcalendar.ProgrammaticLogin() # read calendar.ics file from Evolution cal = Calendar.from_ical(open(conf["CALENDAR_FILE"], 'rb').read()) for event in cal.walk(name="vevent"): created_ds = str_to_dttime(event.get("created").to_ical()) if created_ds < start_ds or created_ds > end_ds: continue start = event.get("dtstart").to_ical() end = event.get("dtend").to_ical() summary = event.get("summary") if summary == None: summary = "None" description = event.get("description") if description == None: description = "None" location = event.get("location") if location == None: location = "None" # create gdata event and populate gevent = gdata.calendar.CalendarEventEntry() gevent.title = atom.Title(text=summary.encode("utf-8")) gevent.content = atom.Content(text=description.encode("utf-8")) gevent.where.append(gdata.calendar.Where( value_string=location.encode("utf-8"))) rrule = event.get("rrule") if rrule != None: rrule = rrule.to_ical(); recurrence_data = "DTSTART;VALUE=DATE:" + start + "\r\n" + \ "DTEND;VALUE=DATE:" + end + "\r\n" + \ "RRULE:" + rrule + "\r\n" gevent.recurrence = gdata.calendar.Recurrence(text=recurrence_data) else: start_z = dttime_to_iso_zulu(str_to_dttime(start)) end_z = dttime_to_iso_zulu(str_to_dttime(end)) gevent.when.append(gdata.calendar.When( start_time=start_z, end_time=end_z)) gcalendar.InsertEvent(gevent, conf["GOOGLE_CALENDAR_FEED_URL"]) print "Inserted event: %s (%s-%s)" % (summary, start, end) if __name__ == "__main__": main()

On the GCalendar side, I set up notifications for every calendar event so that it sends me an SMS 5 minutes before the event.

And thats pretty much it. Took me couple of weeks of reading, fiddling and coding to get this done, but now I have Email and Calendaring on my Android that are as good as on the Blackberry.