更新
2019/1/16：增加了允许5个附件发送、smtp等配置的字段。
一、背景
在ETL数据整合过程中，肯定会涉及到email的通知，比如ETL的执行情况汇报，执行耗时汇报，关键数据更新情况等信息汇报等，这些信息都是需要及时给到相应的operation人员或者使用BI数据的人员。
但是，如果一开始没有规划好邮件推送的一些基本信息，有可能会导致后期邮件发送混乱和不好管理等问题，例如：每个人都有自己的etl，每个人都会去开发自己的邮件通知，那随着时间推移，后期哪些邮件要取消、哪些通知人要屏蔽等都是个难事，可能需要打开所有的ETL job去检查，去修改，耗时耗力，非常不利于管理。
二、实现
在实现该方案的时候，我主要考虑了以下几个方面：
1、每个人需要调用发送邮件的时候，尽量不要重复再做一次拖拉整套组件了，拖拉一次公共组件就好了，所以我选择用joblet来实现这个。
2、邮件的一些基础公共信息必须在一个地方维护，比如发送、接收邮件列表，发送记录等信息，所以我设计了数据库表来存放这些信息，这样只要更新数据库信息，就可以使得全局都使用统一的信息。
3、信息的发送、状态、生成的方式都可以灵活控制，所以我设计了一个表来存储这些信息，而且通过存储过程生成具体的email信息，这样可以追踪发送记录等信息。
4、因为talend joblet支持变量，所以我尽量将发送邮件组件中的一些变量都设计到数据库表中，这样方便于维护和修改。
2.1、数据库表设计
数据库设计主要有2张表：mail_send_group、mail_send_list_rec
mail_send_group：该表是用于记录发送者和接收者之间的信息，维护在这里可以让后去维护更简单，修改数据库则全局启用。
IF (OBJECT_ID(N'[chk].[mail_send_group]', N'U') IS NOT NULL) BEGIN PRINT N'删除表：[chk].[mail_send_group]'; DROP TABLE [chk].[mail_send_group]; END GO CREATE TABLE [chk].[mail_send_group] ( [group_id] NVARCHAR(50) NOT NULL,--主键 [mail_to] NVARCHAR(1000) NOT NULL,--接收者邮箱列表，多个邮箱用;分割 [mail_from] NVARCHAR(100) NOT NULL,--发送者邮箱 [sender_name] NVARCHAR(100) NOT NULL,--发送者昵称 [mail_cc] NVARCHAR(1000) NULL,--抄送邮箱列表，多个邮箱用;分割 [mail_bcc] NVARCHAR(100) NULL,--密送邮箱列表，多个邮箱用;分割 [smtp_host] NVARCHAR(100) NOT NULL,--smtp host地址 [smtp_port] INT NOT NULL,--smtp host 端口号 [user_name] NVARCHAR(50) NOT NULL,--邮箱用户名 [user_pwd] NVARCHAR(50) NOT NULL,--邮箱用户密码 [create_date] DATETIME NOT NULL,--创建日期 [status] SMALLINT NULL--状态(0禁用，1启用) ) GO SELECT * FROM [chk].[mail_send_group] |
mail_send_list_rec：该表是用于记录email生成的记录和发送记录的，每条信息通过group_id和上表关联，就可以知道每条信息是由谁发给谁的，什么时候发送的。
IF (OBJECT_ID(N'[chk].[mail_send_list_rec]', N'U') IS NOT NULL) BEGIN PRINT N'删除表：[chk].[mail_send_list_rec]'; DROP TABLE [chk].[mail_send_list_rec]; END GO CREATE TABLE [chk].[mail_send_list_rec] ( [mail_id] NVARCHAR(50) NOT NULL,--主键 [group_id] NVARCHAR(50) NOT NULL,--所属的group id，用于确认发送接收等信息 [scope] NVARCHAR(100) NOT NULL,--业务，用于区分不同业务生成的邮件，相当于一个分类 [subject] NVARCHAR(100) NOT NULL,--主题 [message] NVARCHAR(4000) NOT NULL,--正文，支持HTML代码，建议是用HTML [create_date] DATETIME NOT NULL,--创建日期 [send_date] DATETIME NULL,--发送日期 [send_status] SMALLINT NULL,--发送状态（0创建完未发送，1已经发送） [atta01_path] NVARCHAR(200) NULL,--第1个发送附件路径绝对路径，包含文件名/data/mailatts/ids/checkret.csv [atta02_path] NVARCHAR(200) NULL,--第1个发送附件路径绝对路径，包含文件名 [atta03_path] NVARCHAR(200) NULL,--第1个发送附件路径绝对路径，包含文件名 [atta04_path] NVARCHAR(200) NULL,--第1个发送附件路径绝对路径，包含文件名 [atta05_path] NVARCHAR(200) NULL--第1个发送附件路径绝对路径，包含文件名 ) GO |
2.2、Joblet开发
1、 Joblet采用了input方式，输入的参数是mail_id，即邮件的ID，这个是外部调用这个joblet的时候需要传递进来的一个参数。
2、 tFlowToIterate是用于将mail_id生成一个全局变量，用于传递给3的mssqlinput。
3、 该组件是用于根据mail_id去查询数据库表中的详细email信息，为后面的发送信息提供明细。
"SELECT [a].[mail_id] ,[a].[subject] ,[a].[message] ,[b].[mail_from] ,[b].[mail_to] ,[b].[sender_name] ,[b].[mail_cc] ,[b].[mail_bcc] ,[b].[status] ,[a].[atta01_path] ,[a].[atta02_path] ,[a].[atta03_path] ,[a].[atta04_path] ,[a].[atta05_path] ,[b].[smtp_host] ,[b].[smtp_port] ,[b].[user_name] ,[b].[user_pwd] FROM [chk].[mail_send_list_rec] AS a INNER JOIN [chk].[mail_send_group] AS b ON ([a].[group_id] = [b].[group_id]) WHERE [a].[mail_id] = '" + ((String)globalMap.get("curr_mail_id")) + "' AND ISNULL([b].[status], 0) = 1" |
4、 发送邮件组件主要就是根据数据库中查询的数据，通过变量方式传递过来后，执行发送邮件的操作。
5、 更新数据库中相应的mail_id的记录为已发送和发送时间等信息。先用tfixedflowinput生成相应的存储过程参数，然后MSSQL_SP调用存储过程更新。
2.3、存储过程开发生成&更新email内容
生成email：主要功能就是按照你想要发送的内容生成一个message，并插入到数据库表中即可。
IF (OBJECT_ID(N'[chk].[usp_insert_ids_mail_send_list_rec]', N'P') IS NOT NULL) BEGIN PRINT N'删除存储过程：[chk].[usp_insert_ids_mail_send_list_rec]'; DROP PROC [chk].[usp_insert_ids_mail_send_list_rec]; END GO CREATE PROC [chk].[usp_insert_ids_mail_send_list_rec] ( @curr_date NVARCHAR(20), @atta01_path NVARCHAR(200), @atta02_path NVARCHAR(200), @atta03_path NVARCHAR(200), @atta04_path NVARCHAR(200), @atta05_path NVARCHAR(200) ) AS --==================================================================================================================================== -- ProcedureName : chk.usp_insert_ids_mail_send_list_rec -- Author : john.xiong -- CreateDate : 2019-01-02 -- Description : 生成daily的detail mail content /*************************************Parameters参数说明******************************************************************************* -- @curr_date : 数据实行日期YYYYMMDD **************************************Modfied List修改记录***************************************************************************** -- Modified Date Modified User Version Modified Reason ************************************************************************************************************************************** -- 2019-01-02 john.xiong V01.00.00 初始化版本 **************************************************************************************************************************************/ --==================================================================================================================================== BEGIN BEGIN TRY DECLARE @begin_time DATETIME ,@end_time DATETIME ,@cost_time INT; SET @begin_time = DATEADD(HOUR, 8, GETDATE()); INSERT INTO [chk].[tb_proc_cost_log] ( [proc_name] ,[Object_name] ,[execute_time] ,[action] ,[remark] ,[cost_time] ) SELECT N'chk.usp_insert_ids_mail_send_list_rec' AS [proc_name] ,N'chk.mail_send_list_rec' AS [Object_name] ,@begin_time AS [execute_time] ,N'start' AS [action] ,'' AS [remark] ,0 AS [cost_time] DECLARE @mail_id UNIQUEIDENTIFIER, @scope NVARCHAR(100), @group_id UNIQUEIDENTIFIER, @subject NVARCHAR(100), @create_date DATETIME, @message NVARCHAR(4000), @temp_message NVARCHAR(4000), @count INT, @count1 INT, @count2 INT, @error_count INT SET @mail_id = NEWID(); SET @scope = N'IDS'; SET @group_id = N'8D42D25D-59C7-4A5E-AE9C-4A5F24D910B0' SET @subject = N'IDS daily - job运行情况'; SET @create_date = DATEADD(HOUR, 8, GETDATE()); SET @count1 = 0; SET @count2 = 0; SET @error_count = 0; SET @message = '<span style="color:#000; line-height:30px"><ol>'; SET @temp_message = ''; SET @count = 0; SELECT @count = COUNT(*) FROM [chk].[log_move_blob_rec] AS a WHERE LEFT([a].[rec_load_time], 8) = @curr_date AND ([a].[scope] IN ('ids_regular_data', 'ids_regular_rtm') OR [a].[blobFileName] LIKE '%LCH%') SET @message = @message + N'<li>从landing搬移blob文件总数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)); SET @temp_message = ''; SET @count = 0; SELECT @count = COUNT(*) FROM [chk].[log_move_blob_rec] AS a WHERE LEFT([a].[rec_load_time], 8) = @curr_date AND [a].[scope] = 'ids_regular_data' SET @message = @message + N'<br>经销商regular data文件数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)); SET @temp_message = ''; SET @count = 0; SELECT @count = COUNT(*) FROM [chk].[log_move_blob_rec] AS a WHERE LEFT([a].[rec_load_time], 8) = @curr_date AND [a].[scope] IN ('ids_regular_rtm') SET @message = @message + N'<br>restatement data文件数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)); SET @temp_message = ''; SET @count = 0; SELECT @count = COUNT(*) FROM [chk].[log_move_blob_rec] AS a WHERE LEFT([a].[rec_load_time], 8) = @curr_date AND [a].[blobFileName] LIKE '%LCH%' SET @message = @message + N'<br>local customer hierarchy daily文件数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)) + '</li>'; SET @temp_message = ''; SET @count = 0; SELECT @count = SUM([a].[file_count]) FROM [chk].[log_blob_file_deal] AS a WHERE LOWER([a].[data_scope]) = 'ids' AND LOWER([a].[deal_level]) = 'ext' AND LOWER([a].[job_name]) = LOWER('IDS_Data_Blob_To_Stg_Ongoing_Loop_New_1_3') AND [a].[remark] LIKE '%tFileList Count%' AND CONVERT(NVARCHAR(8), [a].[deal_date], 112) = @curr_date SET @message = @message + N'<li>实际处理经销商regular data文件数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)) + '</li>'; SET @temp_message = ''; SET @count = 0; SELECT @count = SUM([a].[file_count]) FROM [chk].[log_blob_file_deal] AS a WHERE LOWER([a].[data_scope]) = 'ids' AND LOWER([a].[deal_level]) = 'ext' AND LOWER([a].[job_name]) = LOWER('IDS_Data_Blob_To_Stg_SalesDaily_Rtm_New_1_4') AND [a].[remark] LIKE '%tFileList Count%' AND CONVERT(NVARCHAR(8), [a].[deal_date], 112) = @curr_date SET @message = @message + N'<li>实际处理restatement data文件数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)) + '</li>'; SET @temp_message = ''; SET @count = 0; SELECT @count = COUNT([a].[file_name]) FROM [chk].[log_file_deal_error_rec] AS a WHERE LOWER([a].[data_scope]) = 'ids' AND LOWER([a].[deal_level]) = 'ext' AND LOWER([a].[job_name]) = LOWER('IDS_Data_Blob_To_Stg_Ongoing_Loop_New_1_3') AND CONVERT(NVARCHAR(8), [a].[deal_date], 112) = @curr_date SET @message = @message + N'<li>无法解压的经销商regular data文件数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)) + '</li>'; SET @temp_message = ''; SET @count = 0; SELECT @count = COUNT([a].[file_name]) FROM [chk].[log_file_deal_error_rec] AS a WHERE LOWER([a].[data_scope]) = 'ids' AND LOWER([a].[deal_level]) = 'ext' AND LOWER([a].[job_name]) = LOWER('IDS_Data_Blob_To_Stg_SalesDaily_Rtm_New_1_4') AND CONVERT(NVARCHAR(8), [a].[deal_date], 112) = @curr_date SET @message = @message + N'<li>无法解压的restatement data文件数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)) + '</li>'; SET @temp_message = ''; SET @count = 0; SELECT @count = SUM([a].[file_count]) FROM [chk].[log_blob_file_deal] AS a WHERE LOWER([a].[data_scope]) = 'ids' AND LOWER([a].[deal_level]) = 'ext' AND LOWER([a].[job_name]) = LOWER('IDS_RCS_Local_Master_Data_Daily_1_2') AND [a].[remark] LIKE '%tFileList Count lch%' AND CONVERT(NVARCHAR(8), [a].[deal_date], 112) = @curr_date SET @message = @message + N'<li>处理local customer hierarchy daily文件数：' + CONVERT(NVARCHAR(20), ISNULL(@count, 0)); SET @temp_message = ''; SET @count = 0; SET @count1 = 0; SELECT TOP (1) @count1 = [a].[row_count] FROM [chk].[log_table_data_rec] AS a WHERE [a].[data_scope] = 'rcs dim' AND [a].[table_name] = 'stg.cust_ids_rcs_local_customer_hierarchy_daily' AND CONVERT(NVARCHAR(8), [a].[action_time], 112) = @curr_date ORDER BY [a].[action_time] DESC SET @message = @message + N'<br>文件数据行数：' + CONVERT(NVARCHAR(20), ISNULL(@count1, 0)); SET @temp_message = ''; SET @count = 0; SET @count2 = 0; SELECT @count2 = COUNT(*) FROM [stg].[cust_ids_rcs_local_customer_hierarchy_daily] AS a WHERE LEFT([a].[rec_load_time], 8) = @curr_date SET @message = @message + N'<br>入库数据行数：' + CONVERT(NVARCHAR(20), ISNULL(@count2, 0)) + '</li>'; IF (@count1 <> @count2) BEGIN SET @error_count = @error_count + 1; END IF (OBJECT_ID(N'[chk].[temp_mail_send_proc_error_list_ids_daily]', N'U') IS NOT NULL) BEGIN DROP TABLE [chk].[temp_mail_send_proc_error_list_ids_daily]; END /*生成错误proc的记录*/ CREATE TABLE [chk].[temp_mail_send_proc_error_list_ids_daily] WITH ( DISTRIBUTION = ROUND_ROBIN, CLUSTERED COLUMNSTORE INDEX ) AS SELECT [a].[proc_name] ,ROW_NUMBER() OVER(ORDER BY [a].[error_time] ASC) AS [Num] FROM [chk].[log_proc_error_rec] AS a WHERE [a].[proc_name] LIKE '%ids%' AND [a].[proc_name] NOT LIKE '%mail%' AND CONVERT(NVARCHAR(8), [a].[error_time], 112) = @curr_date SET @count = 0; SELECT @count = COUNT(*) FROM [chk].[temp_mail_send_proc_error_list_ids_daily]; IF (@count > 0) BEGIN SET @message = @message + N'<li style="color:red">有错误的PROC：' + CONVERT(NVARCHAR(20), @count); SET @error_count = @error_count + @count; END WHILE (@count > 0) BEGIN SELECT @temp_message = [proc_name] FROM [chk].[temp_mail_send_proc_error_list_ids_daily] WHERE [Num] = @count; SET @message = @message + N'<br />' + @temp_message + '; '; SET @count = @count - 1; END SET @message = @message + '</li>'; IF (@error_count <> 0) BEGIN SET @subject = @subject + '：有 ' + CONVERT(NVARCHAR(20), @error_count) + ' 个错误'; END SET @subject = @curr_date + N' ' + @subject; SET @message = @message + '</ol></span>' PRINT @message INSERT INTO [chk].[mail_send_list_rec] ( [mail_id] ,[group_id] ,[scope] ,[subject] ,[message] ,[create_date] ,[send_date] ,[send_status] ,[atta01_path] ,[atta02_path] ,[atta03_path] ,[atta04_path] ,[atta05_path] ) SELECT @mail_id, @group_id, @scope, @subject, @message, @create_date, NULL, 0, @atta01_path, @atta02_path, @atta03_path, @atta04_path, @atta05_path SET @end_time = DATEADD(HOUR, 8, GETDATE()); SET @cost_time = DATEDIFF(SECOND, @begin_time, @end_time); INSERT INTO [chk].[tb_proc_cost_log] ( [proc_name] ,[Object_name] ,[execute_time] ,[action] ,[remark] ,[cost_time] ) SELECT N'chk.usp_insert_ids_mail_send_list_rec' AS [proc_name] ,N'chk.mail_send_list_rec' AS [Object_name] ,@end_time AS [execute_time] ,N'end' AS [action] ,CONVERT(NVARCHAR(50), @mail_id) AS [remark] ,@cost_time AS [cost_time] PRINT N'Exec success'; SELECT @mail_id AS [curr_mail_id] END TRY BEGIN CATCH INSERT INTO [chk].[log_proc_error_rec] ( [proc_name] ,[error_source] ,[error_time] ,[error_severity] ,[error_state] ,[error_msg] ,[log_user] ) SELECT N'chk.usp_insert_ids_mail_send_list_rec' AS [proc_name] ,ERROR_PROCEDURE() AS [error_source] ,DATEADD(HOUR, 8, GETDATE()) AS [error_time] ,ERROR_SEVERITY() AS [error_severity] ,ERROR_STATE() AS [error_state] ,ERROR_MESSAGE() AS [error_msg] ,SUSER_SNAME() AS [log_user] PRINT N'Exec failed'; END CATCH END |
更新email by mail_id
IF (OBJECT_ID(N'[chk].[usp_update_mail_send_list_rec_by_mail_id]', N'P') IS NOT NULL) BEGIN PRINT N'删除存储过程：[chk].[usp_update_mail_send_list_rec_by_mail_id]'; DROP PROC [chk].[usp_update_mail_send_list_rec_by_mail_id]; END GO CREATE PROC [chk].[usp_update_mail_send_list_rec_by_mail_id] ( @mail_id NVARCHAR(50) ,@send_date DATETIME ,@send_status SMALLINT ) AS --==================================================================================================================================== -- ProcedureName : [chk].[usp_update_mail_send_list_rec_by_mail_id] -- Author : john.xiong -- CreateDate : 2018-12-24 -- Description : 根据mail_id更新mail发生记录信息 /*************************************Parameters参数说明******************************************************************************* -- @mail_id : 邮件id NEWID **************************************Modfied List修改记录***************************************************************************** -- Modified Date Modified User Version Modified Reason ************************************************************************************************************************************** -- 2018-12-24 john.xiong V01.00.00 初始化版本 **************************************************************************************************************************************/ --==================================================================================================================================== BEGIN BEGIN TRY DECLARE @begin_time DATETIME ,@end_time DATETIME ,@cost_time INT SET @begin_time = DATEADD(HOUR, 8, GETDATE()); INSERT INTO [chk].[tb_proc_cost_log] ( [proc_name] ,[Object_name] ,[execute_time] ,[action] ,[remark] ,[cost_time] ) SELECT N'chk.usp_update_mail_send_list_rec_by_mail_id' AS [proc_name] ,N'chk.mail_send_list_rec' AS [Object_name] ,@begin_time AS [execute_time] ,N'start' AS [action] ,N'' AS [remark] ,0 AS [cost_time] IF (@mail_id IS NULL) BEGIN RAISERROR (N'mail id错误！强制退出', 16, 1); END IF (@send_date IS NULL) BEGIN SET @send_date = DATEADD(HOUR, 8, GETDATE()); END UPDATE [chk].[mail_send_list_rec] SET [send_date] = @send_date, [send_status] = @send_status WHERE [mail_id] = @mail_id; SET @end_time = DATEADD(HOUR, 8, GETDATE()); SET @cost_time = DATEDIFF(SECOND, @begin_time, @end_time); INSERT INTO [chk].[tb_proc_cost_log] ( [proc_name] ,[Object_name] ,[execute_time] ,[action] ,[remark] ,[cost_time] ) SELECT N'chk.usp_update_mail_send_list_rec_by_mail_id' AS [proc_name] ,N'chk.mail_send_list_rec' AS [Object_name] ,@end_time AS [execute_time] ,N'end' AS [action] ,N'' AS [remark] ,@cost_time AS [cost_time] PRINT N'exec successed' END TRY BEGIN CATCH INSERT INTO [chk].[log_proc_error_rec] ( [proc_name] ,[error_source] ,[error_time] ,[error_severity] ,[error_state] ,[error_msg] ,[log_user] ) SELECT N'chk.usp_update_mail_send_list_rec_by_mail_id' AS [proc_name] ,ERROR_PROCEDURE() AS [error_source] ,DATEADD(HOUR, 8, GETDATE()) AS [error_time] ,ERROR_SEVERITY() AS [error_severity] ,ERROR_STATE() AS [error_state] ,ERROR_MESSAGE() AS [error_msg] ,SUSER_SNAME() AS [log_user] PRINT N'exec failed' END CATCH END |
三、和job结合调用
在需要发送email的job中，将joblet拖拉过去即可，然后生成一个你需要发送的邮件的mail_id，通过input组件将其传递到joblet组件的input输入中，这样就可以将joblet融入到job中。
如果您觉得此文章对您有帮助，请点击右下方【推荐】让更多人看到，thanks！