Callbacks from Mandrill

[ACDC]

ACDC
Actually, its easier than I thought.
As shown above, you don't need all the crap from Mandrill.
Once I parse out the single ID field, then its a simple GET function to send the Logon string PLUS your variable to logonOp.awsp

That's great, I don't have a paid MailChimp account and you need one to sign up with Mandrill. So I am in the dark as to the Mandrill Call back setup.
The concept that I have in my mind right now is:

1. Send an Email via the Mandrill SMPT relay server and insert the OutgoingEmail.ID into the email header using the AIM Eheader function (It appears Mandrill have a host of options when using their SMTP Custom Header Reference- "https://mandrill.zendesk.com/hc/en-us/a ... -reference")

2. Receive the event call back via the Node.js as you suggested, or first prize for me, receive the Mandrill call back directly into AIM

When I get some time I will get onto it.

[Jaymer]

The concept that I have in my mind right now is:

1. Send an Email via the Mandrill SMPT relay server and insert the OutgoingEmail.ID into the email header using the AIM Eheader function (It appears Mandrill have a host of options when using their SMTP Custom Header Reference- "https://mandrill.zendesk.com/hc/en-us/a ... -reference")

Tom Ford is already doing this part. Perhaps he can add his experience.

[Jaymer]

The concept that I have in my mind right now is:

2. Receive the event call back via the Node.js as you suggested, or first prize for me, receive the Mandrill call back directly into AIM

RE: receive the Mandrill Webhook directly into AIM

TL;DR This isn't even possible now

ok, do this simple thing.
copy this code below to clipboard.

Code: Select all

[{"event":"open","ts":1536132312,"user_agent":"Mozilla\\/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit\\/537.36 (KHTML, like Gecko) Chrome\\/68.0.3440.84 Safari\\/537.36","user_agent_parsed":{"type":"Browser","ua_family":"Chrome","ua_name":"Chrome 68.0.3440.84","ua_version":"68.0.3440.84", "ua_url":"http:\\/\\/www.google.com\\/chrome","ua_company":"Google Inc.","ua_company_url":"http:\\/\\/www.google.com\\/","ua_icon":"http:\\/\\/cdn.mandrill.com\\/img\\/email-client-icons\\/chrome.png","os_family":"OS X","os_name":"OS X","os_url":"http:\\/\\/www.apple.com\\/osx\\/","os_company":"Apple Computer, Inc.","os_company_url":"http:\\/\\/www.apple.com\\/","os_icon":"http:\\/\\/cdn.mandrill.com\\/img\\/email-client-icons\\/macosx.png","mobile":false},"ip":"70.126.141.227","location":{"country_short":"US","country":"United States","region":"Florida","city":"Tampa","latitude":27.9475193024,"longitude":-82.4584274292,"postal_code":"33601","timezone":"-04:00"},"_id":"443082dac81649978b07f944d251c451","msg":{"ts":1536132236,"_id":"443082dac81649978b07f944d251c451","state":"sent","subject":"My Office Blast 3:18am  275 ppl","email":"[email protected]","tags":["MyCustomTag"],"opens":[{"ts":1536132312,"ip":"70.126.141.227","location":"Florida, US","ua":"OS X\\/OS X\\/Chrome\\/Chrome 68.0.3440.84"}],"clicks":[],"smtp_events":[{"ts":1536132273,"type":"sent","diag":"250 Requested mail action okay, completed: id=0MKsCM-1fxSAz1nwu-0000FE","source_ip":"198.2.180.28","destination_ip":"74.208.5.21","size":2797}],"subaccount":"demo","resends":[],"_version":"DgI4TT6W9EDqtvovn3A3IA","metadata":[{"email_id":"834418"}],"sender":"[email protected]","template":null}}]

go here: https://codebeautify.org/jsonviewer
paste code into left box
click Beautify button in center panel
The "prettied" code shows up on the right.

Thats a real view of the JSON structure Aware would have to parse.
Based on what I've seen getting JSON back from REST replies, it would make sense that IF Vlad offered some way to parse a reply, he'd follow the same routine observed in REST replies, which means:
You'd have to create a BO structure in Aware to accommodate this data.
1) BO "TheEvent" (top level name doesn't matter). Fields like event, ts, ip (and others at that 2nd level)
2) The data in location has to go in another BO. A field in BO_TheEvent is named "location" and is an OS (owned single) of BO_location. All those fields "under" location are now fields in BO_location.
Similarly, any other 2nd level field that is an array of other values becomes another BO. BO_user_agent_parsed and BO_msg
(bear with me, i know this is complicated.)
3) all this repeats with BO_msg. Its top-level fields are: state, subject, email, but BOs must be created to hold the child arrays: tags, opens, clicks, etc.

NOTE: If you do NOT want or need any data from user_agent_parsed, for example, (step 2 above), then Aware gracefully doesn't complain and will ignore that entire element path if you don't have that field defined in BO_TheEvent. So its easy to pick & choose which data you want to preserve, BUT even for just 1 field, you have to create the BO (example. If you only needed Location.timezone, you'd still need BO_location and it would have only 1 field in it).

So now you end up having a BO structure that looks very similar to the Beautify layout:
BO_TheEvent (fields event, ts, user_agent, ip)
L__ BO_user_agent_parsed (fields type, ua_family, ua_name)
L__ BO_location (fields country, region, state)
L__ BO_msg (fields ts, state, subject)
...L___ BO_opens
...L___ BO_smtp_events
...L___ BO_meta-data
......L___ BO_email_id *

Once an Event would be Parsed/Received/???, and IF it follows the same JSON REST Reply routine, then it would create rows and populate all these fields automatically. One BO_TheEvent structure for each Webhook sent by whomever.
All of this structure is in place just to handle 1 Webhook, BUT remember they are also individual AWARE BOs. So, I would add a field in BO_email_id for ProcessedYN, or Date_Processed (these will be undefined or a RULE can set them). Then have a Scheduled Event scan the BO_email_id table for any un-Processed records. Finding the OutgoingEmail record for that ID now links you to all existing Aware data. You can now post opened/clicked or tracking data anywhere you want. You can traverse all the way back up to TheEvent to get the type ( BO_email_id.ob_BO_meta_data.ob_BO_msg.ob_BO_TheEvent.event ).

(* NOTE: I have a problem with this description above. From Beautify, note the difference between locations and opens. opens allows for multiple arrays. Its is an Object of Arrays. Locations has only 1 array.
The names of these elements come from a API Spec. Aware matches up the element name with a real field (attribute) in the BO.
Well, we come to a problem if we are trying to design a Structure to handle Mandrill Webhooks. Because meta-data is just like opens, except the NAME of the element varies - its not in a "spec" anywhere. It just means that if you add another piece of metadata, you have to add another BO just to receive the data for that 1 field. OK, I digress, this may not be a big deal, cause you'd have to go back into code anyway to handle that data anyway. )

[Jaymer]

The concept that I have in my mind right now is:

2. Receive the event call back via the Node.js as you suggested, or first prize for me, receive the Mandrill call back directly into AIM

RE: Receive the Mandrill Webhook via the Node.js as you suggested
Extract data in Node.JS and send a GET to Aware logonOP

I may actually try this if I can't swing others to do it all in Node.js.
WIll come back to this if needed.

BTW, ACDC, WTF is logonOp.awsp ?
Seen this here: https://www.awareim.com/forum/viewtopic ... wsp#p42937
but where is the documentation for this?

[Jaymer]

The concept that I have in my mind right now is:

2. Receive the event call back via the Node.js as you suggested, or first prize for me, receive the Mandrill call back directly into AIM

RE: Receive the Mandrill Webhook via the Node.js as you suggested
Process it ALL in Node.JS and do the SQL update there.

TL;DR This IS possible NOW, and its easier than you think
Bruce's demo on calling Jasper reports and Mailing the PDF was my first exposure to Node.js.
It hasn't been exactly easy, but its because of the JS part, not "Node" per se.

Its easy to get a skeleton "listener" file and attach to your db. No worries.
My issues were from parsing Objects and I'm past that now. (thanks bruce)

Installing Node & testing a connection to your SQL database should be under 1 hour.
And now that I know what I know, under an hour to get a Node server script running to process the Webhooks.

Here's my actual code that examines the Webhook data and does the SQL update.

Code: Select all

if ( 'email_id' in  clickEmailNum ) {
		the_EmailID = clickEmailNum[0]["email_id"];
			var request = new sql.Request();
			request.query('UPDATE OutgoingEmail SET Clicked=1 WHERE ID=' + the_EmailID, 
				function (err, recordset) {   if (err) console.log('Error Updating Email.Clicked: ',err) 	});
}

Sure, there's no rules firing based on this.
On my Unsubscribe event, I'm going to also have to update a variable in the ContactMaster.
One ALTERNATIVE, would be to NOT touch the LIVE database data, and just post these transactions in a to-be-processed file and then do them in Aware from the Scheduler. That would be fine too, and provide an audit trail. But I don't see any need to "bother" Aware with thousands of Logins (using LogonOp.aw) just to flip a byte or two when I'm USUALLY just 1 SQL update away from doing the same thing.

[ACDC]

ok, do this simple thing.
copy this code below to clipboard.

I see what you mean, Node.js seems to be a simple way (well if you are a Node.js expert) it has other uses as well, so its' a good idea to get into it

[Jaymer]

Screen Shot 2018-09-06 at 4.54.14 AM.png (50.83 KiB) Viewed 16990 times

Screen Shot 2018-09-07 at 11.29.25 AM.png (17.95 KiB) Viewed 16990 times

Screen Shot 2018-09-07 at 11.31.24 AM.png (29.86 KiB) Viewed 16990 times

[hpl123]

Screen Shot 2018-09-06 at 4.54.14 AM.png

Screen Shot 2018-09-07 at 11.29.25 AM.png

Screen Shot 2018-09-07 at 11.31.24 AM.png

Cool stuff Jaymer, thanks for sharing details about your adventures, appreciated. When can we expect the plugin

[BenHayat]

When can we expect the plugin

You can't write Java plugin with Node.js (Javascript).

[ACDC]

Jaymer , could you summarise please

1. How you are sending the email and ID via Mandrill, SMTP or REST ?
2. Mandrill Call back to your Node.js (how easy is this to setup with Mandrill once I have Node.js installed(I see Mandrill has a connector for Node.js)
3. Node.js connection to AwareIM - are you using REST ? - I am going to have to use LogonOp.aws with parameters - I am still on 7.1

Thanks in advance

[Jaymer]

1. How you are sending the email and ID via Mandrill, SMTP or REST ?
2. Mandrill Call back to your Node.js (how easy is this to setup with Mandrill once I have Node.js installed(I see Mandrill has a connector for Node.js)
3. Node.js connection to AwareIM - are you using REST ? - I am going to have to use LogonOp.aws with parameters - I am still on 7.1

Q: 1. How you are sending the email and ID via Mandrill, SMTP or REST ?
1. per this post: https://www.awareim.com/forum/viewtopic ... 574#p48578
REST, view the code sample, the JSON string "recipient-metadata"

1b. I have spoken to Tom & Mark in the past 24 hours. Both use Mandrill SMTP.
Per Mandril Docs here: https://mandrill.zendesk.com/hc/en-us/a ... m-metadata
this is how you perform the similar task to get data (like Email_ID) sent with the msg.

Issue # 1: For Mark, this 200-byte limit might be a problem. If you're only sending single emails to 1 person, there is no issue. But sending an Email to 100ppl would not be able to accommodate all the metadata to hold the Email_IDs.
Issue # 2: Mark told me that he builds a string of the 100ppl Email Addresses and stuffs them into the BCC field and then sends 1 email to SMTP Mandrill and then it handles sending it to 100ppl. (True, any email host will do this). But since you're trying to track 100 emails, there has to be 100 aware DB records to get the 100 diff. Email_IDs. I'm not sure he's doing that. And then, per Issue # 1, you can't even build a long enough string to hold more than 15-20 emails of tracking meta-data.

Q: 2. Mandrill Call back to your Node.js (how easy is this to setup with Mandrill once I have Node.js installed(I see Mandrill has a connector for Node.js)
2. There is no "connector" needed. There is an online control panel in Mandril for specifying Webhooks.
This is mine: http://mydomain.com:3321/R3/api/md/mdEmailUpdate

Install Node.js - easy
Install other needed Libraries - easy "npm" command
Install my server script - easy - modify for MySQL vs. MSSQL
Launch and it listens on a port for the Webhook, then write to SQL

Q: 3. Node.js connection to AwareIM - are you using REST ? - I am going to have to use LogonOp.aws with parameters - I am still on 7.1
per this post: https://www.awareim.com/forum/viewtopic ... =15#p48596
There is NO connection to Aware.

[Jaymer]

UPDATE Dec 2018
#3 above. Node connection to Aware

Am using Bandwidth for SMS messaging.
SENDING: w.r.t SMS, I don't much care about actual delivery receipt info, so I have my alert level set to "error" when sending. If a msg isn't delivered, I'll get a JSON msg at my Node.js server. Am not processing this yet except to reply with a 200 - OK

INCOMING SMS: When someone sends a msg to one of the Users' phone #s setup in bandwidth, that ReceiveSMS application calls my Node.js server and is processed. An incoming packet looks like this:

Code: Select all

{ messageId: 'm-echiany4pxxsa',
  from: '+18132940000',
  eventType: 'sms',
  text: 'Yes i got it',
  time: '2018-12-19T04:21:44Z',
  to: '+19163470000',
  state: 'received',
  messageUri:
   'https://api.catapult.inetwork.com/v1/users/u-a6gzr3i6avrrq/messages/m-echiajzny4pxxsa',
  applicationId: 'a-je3pu3kw2zsrafifey',
  direction: 'in' }

I'm using info from there to make this URL call to Aware:

Code: Select all

https://mysite.com/R3/logonOp.aw?domain=AA&userName=SMS&password=aaa&testingMode=true&firstCommand=startProcessWithInit,SMS,main,MessageLog&fromNumber=813-294-0000&toNumber=9163470000&body=Yes%20i%20got%20it&status=stage1&direction=inbound

which logs in to Aware, creates the SMS incoming record MessageLog, where a Rule:
IF MessageLOG IS NEW THEN "do stuff"
processes it and links it to the senders account.
It also looks up the recipient (the user) and forwards a SMS msg to a phone if desired.