如何配置

PAI-Rec引擎已經內置了多個召回模板，包括協同過濾（UserCollaborativeFilterRecall），向量召回（HologresVectorRecall），U2I 召回（UserCustomRecall）等等，并且支持 mysql、hologres、OTS(Tablestore)等多個數據源。

召回公共配置一覽

每種召回配置，都會用到公共配置中的一部分，在此統一解釋，單獨的召回配置中則不再贅述。

配置示例：

"RecallConfs" :[    
  {
    "Name": "collaborative_filter",
    "RecallType": "UserCollaborativeFilterRecall",
    "RecallCount": 1000,
  	"RecallAlgo":"",
    "ItemType":"",
    "CacheAdapter":"",
    "CacheConfig":"",
    "CachePrefix":"",
    "CacheTime":0
  }
]

協同過濾（UserCollaborativeFilterRecall）

協同過濾需要有兩張表，一張u2i表，根據user_id獲取item列表，一張i2i表，獲取相似的item，這兩張表的 schema是固定格式的。

u2i 表

i2i 表

協同過濾不同召回源的配置都是類似的，都在UserCollaborativeDaoConf中進行配置

hologres

配置示例：

"RecallConfs" :[    
  {
    "Name": "collaborative_filter",
    "RecallType": "UserCollaborativeFilterRecall",
    "RecallCount": 1000,
    "UserCollaborativeDaoConf": {
      "AdapterType": "hologres",
      "HologresName": "holo-info",
      "User2ItemTable": "u2i_table",
      "Item2ItemTable": "i2i_table",
      "Normalization" : "on"
    }
  }
]

UserCollaborativeDaoConfig：

ots(tablestore)

"RecallConfs" :[
  {
    "Name": "collaborative_filter",
    "RecallType": "UserCollaborativeFilterRecall",
    "RecallCount": 1000,
    "UserCollaborativeDaoConf": {
      "AdapterType": "tablestore",
      "TableStoreName": "tablestore_info",
      "User2ItemTable": "u2i_table",
      "Item2ItemTable": "i2i_table",
      "Normalization" : "on"
    }
  }
]

redis

Redis進行collaborative_filter流程比較特殊，單獨說明。 實際上也是分兩步：

• 根據RedisPrefix + uid構造key ， 查詢U2I列表， 是個string, 支持格式： item_id1,item_id2,item_id3..... 或者 item_id1:score1,item_id2:score,item_id3:score2......

• 然后查詢I2I列表， 根據上步查詢到的多個item_id，使用MGET進行查詢。 I2I的數據也是string , 格式如下：item_id1:score1,item_id2:score2,item_id3:score3......

配置示例：

"RecallConfs" :[
  {
    "Name": "collaborative_filter",
    "RecallType": "UserCollaborativeFilterRecall",
    "RecallCount": 1000,
    "UserCollaborativeDaoConf": {
      "AdapterType": "redis",
      "RedisName": "redis_info",
      "RedisPrefix": "cr_",
      "Normalization" : "on"
    }
  }
]

實時 U2I2I(RealTimeU2IRecall)

獲取數據的思路和協同過濾中的是一樣的，只不過U2I的數據獲取是通過user歷史行為表實時計算的。

user歷史行為表是根據實時日志來同步更新的，這樣的召回是實時的召回。

user歷史行為表

i2i 表

hologres

行為表建表語句：

BEGIN;
CREATE TABLE "sv_rec"."user_behavior_seq" (
  "user_id" text NOT NULL,
  "item_id" text NOT NULL,
  "event" text NOT NULL,
  "play_time" float8  NULL,
  "timestamp" int8 NOT NULL
);
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'orientation', 'column');
call set_table_property('table_name', 'distribution_key', '"user_id"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'clustering_key', '"user_id:asc","timestamp:desc"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'bitmap_columns', '"user_id","event"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'dictionary_encoding_columns', '"user_id:auto","item_id:auto","event"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'time_to_live_in_seconds', '2592000');
comment on table "sv_rec"."user_behavior_seq" is '用戶實時行為序列';
comment on column "sv_rec"."user_behavior_seq"."user_id" is '用戶id';
comment on column "sv_rec"."user_behavior_seq"."item_id" is 'item id';
comment on column "sv_rec"."user_behavior_seq"."event" is '事件類型';
comment on column "sv_rec"."user_behavior_seq"."play_time" is '閱讀時長,播放時長';
comment on column "sv_rec"."user_behavior_seq"."timestamp" is '時間戳,單位秒';
COMMIT;

配置示例：

"RecallConfs" :[    
    {
      "Name": "RealTimeEtrecRecall",
      "RecallType": "RealTimeU2IRecall",
      "RecallCount": 200,
      "RealTimeUser2ItemDaoConf": {
        "UserTriggerDaoConf": {
          "AdapterType": "hologres",
          "HologresName": "holo_info",
          "HologresTableName": "user_behavior_table",
          "WhereClause": "event='xxx'",
          "Limit": 200,
          "EventPlayTime": "playback:5000;playvslide:5000",
          "EventWeight": "playback:1;playvslide:2",
          "WeightExpression": "(-0.2)*((currentTime-eventTime)/3600/24)",
          "WeightMode": "sum",
          "NoUsePlayTimeField": false
        },
        "Item2ItemTable": "i2i_table"
      }
    }
]

RealTimeUser2ItemDaoConf：

向量召回（HologresVectorRecall）

目前向量召回只能使用hologres數據源，向量數據都存在hologres表中

配置示例：

"RecallConfs" :[    
 {
           "Name": "vector_recall",
           "RecallType": "HologresVectorRecall",
           "RecallCount": 100,
           "VectorDaoConf" :{
               "AdapterType": "hologres",
               "HologresName": "holo_info",
               "HologresTableName": "user_embedding_table",
               "KeyField": "user_id",
               "EmbeddingField" :"emb"
           },
           "HologresVectorConf" :{
               "VectorTable" :"item_embedding_table",
               "VectorEmbeddingField" :"emb",
               "VectorKeyField" :"item_id"
           }
        }
]

VectorDaoConf：

HologresVectorConf：

VectorDaoConf里記錄的是user向量信息，表的定義如下

BEGIN;
CREATE TABLE "public"."graphsage_user_embedding" (
 "user_id" text NOT NULL,
 "emb" float4[] NOT NULL,
 "dt" text,
PRIMARY KEY ("user_id")
);
CALL SET_TABLE_PROPERTY('"public"."graphsage_user_embedding"', 'orientation', 'row');
CALL SET_TABLE_PROPERTY('"public"."graphsage_user_embedding"', 'clustering_key', '"user_id:asc"');
CALL SET_TABLE_PROPERTY('"public"."graphsage_user_embedding"', 'time_to_live_in_seconds', '3153600000');
comment on column "public"."graphsage_user_embedding"."user_id" is '用戶ID';
comment on column "public"."graphsage_user_embedding"."emb" is '用戶特征向量';
comment on column "public"."graphsage_user_embedding"."dt" is '日期 yyyyMMdd';
COMMIT;

HologresVectorConf記錄的是item向量， 表定義如下

BEGIN;
CREATE TABLE "public"."graphsage_item_embedding" (
 "item_id" text NOT NULL,
 "emb" float4[] NOT NULL,
PRIMARY KEY ("item_id")
);
CALL SET_TABLE_PROPERTY('"public"."graphsage_item_embedding"', 'orientation', 'column');
CALL SET_TABLE_PROPERTY('"public"."graphsage_item_embedding"', 'bitmap_columns', '"item_id"');
CALL SET_TABLE_PROPERTY('"public"."graphsage_item_embedding"', 'time_to_live_in_seconds', '3153600000');
comment on column "public"."graphsage_item_embedding"."item_id" is '物品ID';
comment on column "public"."graphsage_item_embedding"."emb" is '物品特征向量';
COMMIT;

實時向量召回（OnlineHologresVectorRecall）

實時向量召回和向量召回的實現方式是一致的，也是基于hologres的表數據，不同的是，user向量不是從表里獲取的，而是實時通過模型獲取到的，然后去item向量表查數據。實現思路基本上分為三步：

• 獲取user相關特征，user特征可以從數據表里查詢

• 調用向量模型，獲取user向量。在我們的支持中，模型是部署在EAS上的

• 和向量召回一樣，通過item向量表進行查詢

配置示例：

"RecallConfs" :[    
  {
      "Name": "online_vector_recall",
      "RecallType": "OnlineHologresVectorRecall",
      "RecallCount": 500,
      "UserFeatureConfs": [
        {
          "FeatureDaoConf": {
            "AdapterType": "hologres",
            "HologresName": "holo_info",
            "FeatureKey": "user:uid",
            "UserFeatureKeyName": "userid",
            "HologresTableName": "user_all_feature_table",
            "UserSelectFields": "*",
            "FeatureStore": "user"
          },
          "Features": []
        }
      ],
      "RecallAlgo": "sv_v2_mind",
      "HologresVectorConf": {
        "HologresName": "holo_info",
        "VectorTable": "item_embedding_table",
        "VectorEmbeddingField": "item_emb",
        "VectorKeyField": "item_id"
      }
    }
]

UserFeatureConfs：

HologresVectorConf：

模型sv_v2_mind在AlgoConfs里定義如下

{
"AlgoConfs": [
  {
      "Name": "sv_v2_mind",
      "Type": "EAS",
      "EasConf": {
        "Processor": "EasyRec",
        "Timeout": 100,
        "ResponseFuncName": "easyrecUserEmbResponseFunc",
        "Url": "http://xxx.vpc.cn-beijing.pai-eas.aliyuncs.com/api/predict/sv_v2_mind",
        "Auth": "xxx"
      }
  }
]
}

和排序模型的配置是一樣的，不同點是ResponseFuncName需要固定為easyrecUserEmbResponseFunc

U2I召回

根據user id來找到對應的item列表。這里的表定義是約定好的

u2i 表

hologres

配置示例：

"RecallConfs" :[    
{
           "Name": "user2item_recall",
           "RecallType": "UserCustomRecall",
           "RecallCount": 500,
           "DaoConf" :{
               "AdapterType": "hologres",
               "HologresName": "holo_info",
               "HologresTableName": "user_item_table"
           }
}
]

OTS(Tablestore)

配置示例：

"RecallConfs" :[    
{
           "Name": "user2item_recall",
           "RecallType": "UserCustomRecall",
           "RecallCount": 500,
           "DaoConf" :{
               "AdapterType": "tablestore",
               "TableStoreName": "ots_info",
               "TableStoreTableName": "user_item_table"
           }
}
]

Redis

配置示例：

"RecallConfs" :[    
{
           "Name": "user2item_recall",
           "RecallType": "UserCustomRecall",
           "RecallCount": 500,
           "DaoConf" :{
               "AdapterType": "redis",
               "RedisName": "redis_info", 
               "RedisPrefix": ""
           }
}
]

圖召回（GraphRecall）

圖召回也屬于U2I召回的一種。通過GraphCompute圖數據庫進行召回。GraphCompute文檔參考概覽。

配置示例：

"RecallConfs" :[    
{
  "Name": "graph_recall",
  "RecallType": "GraphRecall",
  "RecallCount": 500,
  "GraphConf": {
    "GraphName": "graph_test",
    "ItemId": "item_id",
    "QueryString": "g(\"test\").V(\"$1\").hasLabel(\"user\").outE().inV()",
    "Params": ["user.uid"]
  }
}
]

GraphConf：

用戶分組熱門召回（UserGroupHotRecall）

分組熱門召回的表的格式也是約定好的

group_hot_table

按照用戶特征和context信息（例如地域、機型等）組裝trigger_id

• 按照順序將特征用下劃線（_）拼接為trigger_id

• 特征值為空對應 "NULL"

• 包含 Boundaries 字段的特征需要進行離散化（左開，右閉區間），比如年齡 [20, 30, 40, 50] --> trigger對應 <=20, 20-30, 30-40，40-50, >50

用戶年齡23，對應"20-30"

用戶年齡空，對應"NULL"

用戶年齡60，對應">50"

用戶年齡19，對應"<=20"

hologres表示例：

此處使用性別、年齡、機型三個特征

配置示例：

"RecallConfs" :[    
{
          "Name":"user_group_hot_recall",
          "RecallType": "UserGroupHotRecall",
          "RecallCount" :500,
          "Triggers": [
            {
              "TriggerKey": "gender"
            },
             {
              "TriggerKey": "age",
              "Boundaries": [20,30,40,50]
            },
            {
              "TriggerKey": "os"
            }
          ],
          "DaoConf":{
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "group_hotness_table"
          }
}
]

全局熱門召回（UserGlobalHotRecall）

全局熱門召回的表schema和分組召回的表schema是相同的，只是全局熱門召回表中只有一條數據，而且 trigger_id = -1

hologres

配置示例：

"RecallConfs" :[    
{
          "Name":"UserGlobalHotRecall",
          "RecallType": "UserGlobalHotRecall",
          "RecallCount" :500,
          "DaoConf":{
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "global_hotness_table"
          }
  }
]

OTS(Tablestore)

配置示例：

"RecallConfs" :[    
{
          "Name":"UserGlobalHotRecall",
          "RecallType": "UserGlobalHotRecall",
          "RecallCount" :500,
          "DaoConf":{
                "AdapterType": "tablestore",
                "TableStoreName": "ots_info",
                "TableStoreTableName": "global_hotness_recall"
          }
  }
]

冷啟動召回（ColdStartRecall）

查詢item表，根據條件或者時間進行過濾，查詢出符合規則的候選集

hologres

"RecallConfs" :[    
    {
      "Name": "AllLiveItemRecall",
      "RecallType": "ColdStartRecall",
      "RecallCount": 3000,
      "ColdStartDaoConf": {
        "AdapterType": "hologres",
        "HologresName": "holo_info",
        "HologresTableName": "item_status_table",
        "WhereClause": "islist_status=1",
        "PrimaryKey": "\"item_id\"",
        "TimeInterval": 0
      }
    }
]

ColdStartDaoConf：

如何使用

召回的使用位置對應配置總覽中的SceneConfs，SceneConfs是一個Map[string]object結構，可以分場景的使用召回，配置如下

"SceneConfs": {
  "scene_name": {
    "default": {
      "RecallNames": [
        "collaborative_filter"
      ]
    }
  }
}

• scene_name需要替換為自己的場景名。

• default為目錄，這里保持默認即可。

• RecallNames是一個[]string，值為召回配置中的自定義名稱。