Add rake email functionality to get incoming emails from Gmail API (#12231)

* Add rake email functionality to get incoming emails from Gmail API * Fix default max_emails in help menu * Apply suggestions from code review Don't require google api gems in Gemfile, omit redundant `logger.debug` check in gmail.rb Co-authored-by: Oliver Günther <mail@oliverguenther.de> * Add setup instructions and info about Gmail API Rake task to documentation * Remove 'google-apis-core' causing tests to fail --------- Co-authored-by: Joel Giovinazzo <joelgiovinazzo@gmail.com>
2026-06-13 19:20:00 +00:00 · 2023-03-09 07:27:27 +01:00
parent a486f97aec
commit 9302b8ec70
5 changed files with 167 additions and 2 deletions
@@ -313,6 +313,10 @@ gem 'roar', '~> 1.2.0'
 # CORS for API
 gem 'rack-cors', '~> 1.1.1'

+# Gmail API
+gem 'google-apis-gmail_v1', require: false
+gem 'googleauth', require: false
+
 # Required for contracts
 gem 'disposable', '~> 0.6.2'

@@ -479,6 +479,24 @@ GEM
      i18n (>= 0.7)
      multi_json
      request_store (>= 1.0)
+    google-apis-core (0.11.0)
+      addressable (~> 2.5, >= 2.5.1)
+      googleauth (>= 0.16.2, < 2.a)
+      httpclient (>= 2.8.1, < 3.a)
+      mini_mime (~> 1.0)
+      representable (~> 3.0)
+      retriable (>= 2.0, < 4.a)
+      rexml
+      webrick
+    google-apis-gmail_v1 (0.25.0)
+      google-apis-core (>= 0.11.0, < 2.a)
+    googleauth (1.3.0)
+      faraday (>= 0.17.3, < 3.a)
+      jwt (>= 1.4, < 3.0)
+      memoist (~> 0.16)
+      multi_json (~> 1.11)
+      os (>= 0.9, < 2.0)
+      signet (>= 0.16, < 2.a)
    grape (1.7.0)
      activesupport
      builder
@@ -531,6 +549,7 @@ GEM
    json_spec (1.1.5)
      multi_json (~> 1.0)
      rspec (>= 2.0, < 4.0)
+    jwt (2.7.0)
    ladle (1.0.1)
      open4 (~> 1.0)
    launchy (2.5.2)
@@ -566,6 +585,7 @@ GEM
      net-smtp
    marcel (1.0.2)
    matrix (0.4.2)
+    memoist (0.16.2)
    messagebird-rest (1.4.2)
    meta-tags (2.18.0)
      actionpack (>= 3.2.0, < 7.1)
@@ -617,6 +637,7 @@ GEM
      webfinger (>= 1.0.1)
    openproject-token (2.2.0)
      activemodel
+    os (1.1.4)
    paper_trail (12.3.0)
      activerecord (>= 5.2)
      request_store (~> 1.1)
@@ -847,6 +868,11 @@ GEM
    shoulda-context (2.0.0)
    shoulda-matchers (5.3.0)
      activesupport (>= 5.2.0)
+    signet (0.17.0)
+      addressable (~> 2.8)
+      faraday (>= 0.17.5, < 3.a)
+      jwt (>= 1.5, < 3.0)
+      multi_json (~> 1.10)
    spreadsheet (1.3.0)
      ruby-ole
    spring (4.1.1)
@@ -919,6 +945,7 @@ GEM
      addressable (>= 2.8.0)
      crack (>= 0.3.2)
      hashdiff (>= 0.4.0, < 2.0.0)
+    webrick (1.8.1)
    websocket (1.2.9)
    websocket-driver (0.7.5)
      websocket-extensions (>= 0.1.0)
@@ -986,6 +1013,8 @@ DEPENDENCIES
  friendly_id (~> 5.5.0)
  fuubar (~> 2.5.0)
  gon (~> 6.4.0)
+  google-apis-gmail_v1
+  googleauth
  grape (~> 1.7.0)
  grape_logging (~> 1.8.4)
  grids!
@@ -19,10 +19,18 @@ The rake task `redmine:email:receive_imap` fetches emails via IMAP and parses th

 **Packaged installation**

+IMAP:
+
 ```bash
 openproject run bundle exec rake redmine:email:receive_imap host='imap.gmail.com' username='test_user' password='password' port=993 ssl=true allow_override=type,project project=test_project
 ```

+Gmail:
+
+```bash
+openproject run bundle exec rake redmine:email:receive_gmail credentials='/path/to/credentials.json' user_id='test_user' query='is:unread label:openproject' allow_override=type,project
+```
+
 **Docker installation**

 The docker installation has a ["cron-like" daemon](https://github.com/opf/openproject/blob/dev/docker/prod/cron) that will imitate the above cron job. You need to specify the following ENV variables (e.g., to your env list file)
@@ -64,9 +72,41 @@ Available arguments that change how the work packages are handled:
 | `unknown_user` | ignore: email is ignored (default), accept: accept as anonymous user, create: create a user account |
 | `allow_override` | specifies which attributes may be overwritten though specified by previous options. Comma separated list |

+**Gmail API**
+
+In order to use the more secure Gmail API method, some extra initial setup in google cloud is required.
+1. Go to https://console.cloud.google.com/
+2. Create new project
+3. Navigate to Enable APIs and Services
+4. Enable the Gmail API
+5. Navigate to the "Credentials" page for the project
+6. Click "Create Credentials" > "Service Account"
+7. Give the service account editor permissions and click "Done"
+8. Click on the new service account, go to the "Keys" tab, and add a new key.
+9. Save the JSON key file
+  ***Note: Do not give anyone access to this JSON file as it contains the private key to your service account!***
+10. Go to https://admin.google.com
+11. Select Security > Access and Data Control > API Controls
+12. Go to "Domain-Wide Delegation"
+13. Add new API Client
+14. Open JSON key file and copy "client_id" number
+15. Enter `https://www.googleapis.com/auth/gmail.modify` into the scopes
+  ***Note: Modify permissions are necessary here to mark emails as read***
+  ***This is so the service account can access all accounts in your Domain***
+
+Available arguments for the Gmail API rake task that specify the email behavior are
+
+|key | description|
+|----|------------|
+| `credentials` | Gmail service account credentials file (JSON) |
+| `username` | Gmail email address |
+| `query` | Gmail search query (https://support.google.com/mail/answer/7190?hl=en) |
+| `read_on_failure` | Mark emails as read even on failure (default: true) |
+| `max_emails` | Max emails to process (default: 1000) |
+
 ## Format of the emails

-Please note: It's important to use the plain text editor of your email client (instead of the HTML editor) to avoid misinterpretations (e.g. for the project name). 
+Please note: It's important to use the plain text editor of your email client (instead of the HTML editor) to avoid misinterpretations (e.g. for the project name).

 ### Work packages

@@ -183,7 +223,7 @@ In case of receiving errors, the application will try to send an email to the us

 - The configuration setting `report_incoming_email_errors` is true (which it is by default)

-  
+

 By returning an email with error details, you can theoretically be leaking information through the error messages. As from addresses can be spoofed, please be aware of this issue and try to reduce the impact by setting up the integration appropriately.

@@ -0,0 +1,67 @@
+require 'google/apis/gmail_v1'
+require 'googleauth'
+
+module Redmine
+  module Gmail
+    class << self
+      def check(gmail_options={}, options={})
+        credentials = gmail_options[:credentials] || ""
+        username = gmail_options[:user_id] || ""
+        query = gmail_options[:query] || ""
+
+        gmail = Google::Apis::GmailV1::GmailService.new
+        gmail.authorization = authenticate(credentials, username)
+
+        gmail.list_user_messages('me', q: query, max_results: gmail_options[:max_emails]).messages.each do |message|
+          receive(message.id, gmail, gmail_options, options)
+        end
+      end
+
+      def authenticate(credentials, user_id)
+        credentials = Google::Auth::ServiceAccountCredentials.make_creds(
+          json_key_io: File.open(credentials),
+          scope: "https://www.googleapis.com/auth/gmail.modify"
+        )
+        credentials.update!(sub: user_id)
+
+        credentials
+      end
+
+      def receive(message_id, gmail, gmail_options, options)
+        email = gmail.get_user_message('me', message_id, format: "raw")
+        msg = email.raw
+
+        raise "Messages was not successfully handled." unless MailHandler.receive(msg, options)
+
+        message_received(message_id, gmail, gmail_options)
+      rescue StandardError => e
+        Rails.logger.error { "Message #{message_id} resulted in error #{e} #{e.message}" }
+        message_error(message_id, gmail, gmail_options)
+      end
+
+      def message_received(message_id, gmail, gmail_options)
+        log_debug { "Message #{message_id} successfully received" }
+
+        modify_request = Google::Apis::GmailV1::ModifyThreadRequest.new(remove_label_ids: ['UNREAD'])
+        gmail.modify_message("me", message_id, modify_request)
+      end
+
+      def message_error(message_id, gmail, gmail_options)
+        log_debug { "Message #{message_id} can not be processed" }
+
+        if gmail_options[:read_on_failure]
+          modify_request = Google::Apis::GmailV1::ModifyThreadRequest.new(remove_label_ids: ['UNREAD'])
+          gmail.modify_message("me", message_id, modify_request)
+        end
+      end
+
+      def log_debug(&)
+        logger.debug(&)
+      end
+
+      def logger
+        Rails.logger
+      end
+    end
+  end
+end
@@ -179,6 +179,31 @@ namespace :redmine do
      Redmine::POP3.check(pop_options, options_from_env)
    end

+    desc <<~END_DESC
+            Read emails from the Gmail API
+      #{'      '}
+            Available Gmail options:
+              credentials=CREDENTIALS_FILE	Gmail Service Account Credentials File (JSON)
+              username=EMAIL			          Email Address
+              query=QUERY			              Gmail Query String
+              read_on_failure=1             Mark email as read on failure
+              max_emails=1000               Max num of emails to process
+      #{'      '}
+            See redmine:email:receive_gmail for more options and examples.
+    END_DESC
+
+    task receive_gmail: :environment do
+      gmail_options = {
+        credentials: ENV.fetch('credentials', nil),
+        user_id: ENV.fetch('user_id', nil),
+        query: ENV.fetch('query', nil),
+        read_on_failure: ActiveRecord::Type::Boolean.new.cast(ENV.fetch('read_on_failure', 1)),
+        max_emails: ENV.fetch('max_emails', 1000)
+      }
+
+      Redmine::Gmail.check(gmail_options, options_from_env)
+    end
+
    desc 'Send a test email to the user with the provided login name'
    task :test, [:login] => :environment do |_task, args|
      login = args[:login]