方案概述

泛時(shí)序數(shù)據(jù)廣泛應(yīng)用于車聯(lián)網(wǎng)、工業(yè)物聯(lián)網(wǎng)、金融交易、股票分析等多個(gè)業(yè)務(wù)場(chǎng)景。隨著業(yè)務(wù)規(guī)模的增長(zhǎng)，數(shù)據(jù)量急劇增加，如何高效獲取和分析這些數(shù)據(jù)成為業(yè)務(wù)洞察和決策的關(guān)鍵挑戰(zhàn)。以車聯(lián)網(wǎng)場(chǎng)景為例，傳統(tǒng)的數(shù)據(jù)處理通常涉及多個(gè)數(shù)據(jù)組件的整合，運(yùn)維難度高、數(shù)據(jù)存儲(chǔ)及開發(fā)成本高。Lindorm作為阿里云自研的云原生多模數(shù)據(jù)庫，不僅具備低成本存儲(chǔ)、彈性高可用的特點(diǎn)，還集數(shù)據(jù)存儲(chǔ)和分析功能于一體，支持存儲(chǔ)車輛行駛軌跡、車輛狀態(tài)、精準(zhǔn)定位等車聯(lián)網(wǎng)關(guān)鍵數(shù)據(jù)，并提供實(shí)時(shí)或批量數(shù)據(jù)處理能力。

本方案以車輛網(wǎng)場(chǎng)景為例，引導(dǎo)您使用云原生多模數(shù)據(jù)庫 Lindorm的多種引擎，實(shí)現(xiàn)泛時(shí)序數(shù)據(jù)的高并發(fā)實(shí)時(shí)查詢和寫入，幫助您高效構(gòu)建車機(jī)數(shù)據(jù)上報(bào)、物流軌跡跟蹤、實(shí)時(shí)定位調(diào)度等場(chǎng)景服務(wù)。

方案架構(gòu)

方案優(yōu)勢(shì)

• 一體化：一體化操作，開發(fā)效率高，多種數(shù)據(jù)一個(gè)入口，統(tǒng)一SQL操作。相比于傳統(tǒng)的多個(gè)產(chǎn)品支撐一個(gè)場(chǎng)景的方案，架構(gòu)更加簡(jiǎn)單，易于開發(fā)。

• 低成本：低成本存儲(chǔ)，支持百PB級(jí)存儲(chǔ)，支持自適應(yīng)壓縮和表級(jí)別的冷熱分離，您可以根據(jù)數(shù)據(jù)處理需求自由選擇行存儲(chǔ)或列存儲(chǔ)，最多可節(jié)省約50%的成本。

• 高性能：高并發(fā)讀寫，支持千萬級(jí)并發(fā)和毫秒級(jí)延遲，可滿足車聯(lián)網(wǎng)大規(guī)模數(shù)據(jù)的在線存儲(chǔ)與查詢需求。豐富的檢索能力，通過寬表引擎和搜索引擎組合，使用SQL一鍵在所需數(shù)據(jù)列上建立搜索（倒排）索引，提供在線搜索服務(wù)。搜索引擎支持多維檢索、全文檢索、地理位置檢索等多種高級(jí)檢索能力。

• 彈性：云原生彈性，靈活擴(kuò)展，Lindorm計(jì)算引擎具備Serverless特性，可按需分配計(jì)算資源，進(jìn)行高效的交互式分析和批處理。

部署流程

適用場(chǎng)景

• 車聯(lián)網(wǎng)：使用Lindorm存儲(chǔ)車聯(lián)網(wǎng)中的行駛軌跡、車輛狀況、精準(zhǔn)定位等重要數(shù)據(jù)，提供低成本、彈性、靈活可靠的能力，幫助您構(gòu)建高效的網(wǎng)約車、物流運(yùn)輸、新能源車檢測(cè)等場(chǎng)景服務(wù)；集成Ganos時(shí)空數(shù)據(jù)庫引擎，高效支撐車聯(lián)網(wǎng)軌跡數(shù)據(jù)時(shí)空分析與挖掘等相關(guān)應(yīng)用場(chǎng)景。

• 物聯(lián)網(wǎng)：使用Lindorm存儲(chǔ)來自工業(yè)物聯(lián)網(wǎng)場(chǎng)景下的海量異構(gòu)IT&OT數(shù)據(jù)，將數(shù)據(jù)進(jìn)行同構(gòu)、匯聚、融合，打破工業(yè)場(chǎng)景IT數(shù)據(jù)和OT數(shù)據(jù)割裂的“信息孤島”；通過Lindorm計(jì)算引擎提供的低成本、高性能、穩(wěn)定可靠的分布式計(jì)算能力，滿足您在數(shù)智化生產(chǎn)、交互式數(shù)據(jù)探索分析、AI/ML數(shù)據(jù)處理和大規(guī)模圖計(jì)算等場(chǎng)景中的計(jì)算需求。

• 交易賬單：使用Lindorm存儲(chǔ)金融交易中的海量訂單記錄，金融風(fēng)控中的用戶事件、畫像特征、規(guī)則模型、設(shè)備指紋等重要數(shù)據(jù)，提供低成本、高并發(fā)、靈活可靠的能力，幫助您構(gòu)建高效的金融交易與風(fēng)控服務(wù)。

創(chuàng)建列存索引

在進(jìn)行復(fù)雜查詢之前，您需要?jiǎng)?chuàng)建列存索引來增強(qiáng)數(shù)據(jù)分析計(jì)算能力，從而實(shí)現(xiàn)高效的數(shù)據(jù)查詢。

1. 通過Lindorm-cli連接寬表引擎。

   ./lindorm-cli -url <jdbc url> -username <username> -password <password>

2. 創(chuàng)建列存索引column_index_dwd_vehi。

   CREATE INDEX column_index_dwd_vehicle_tbox USING COLUMNAR
   ON dwd_vehicle_tbox(*)
   PARTITION BY ENUMERABLE (bucket(128, vin))
   WITH (`lindorm_columnar.user.index.database` = 'vehicle_db', `lindorm_columnar.user.index.table` = 'dwd_vehicle_tbox_column_idx');

3. （可選）驗(yàn)證索引信息和狀態(tài)。

   SHOW INDEX FROM dwd_vehicle_tbox;

   返回結(jié)果：

   +--------------+------------------+-------------------------------+-------------+----------------+------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------+
   | TABLE_SCHEMA |    DATA_TABLE    |          INDEX_NAME           | INDEX_STATE | INDEX_PROGRESS | INDEX_TYPE | INDEX_COVERED |                                                                                                                                                                                                                                                                                         INDEX_COLUMN                                                                                                                                                                                                                                                                                          | INDEX_TTL |            INDEX_DESCRIPTION            |
   +--------------+------------------+-------------------------------+-------------+----------------+------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------+
   | default      | dwd_vehicle_tbox | column_index_dwd_vehicle_tbox | ACTIVE      |                | COLUMNAR   | NA            | vin,sampletime,DriverStateLinkStatus,AirBagSystemStatus,HODHandoffmonitorValid,BodyCtrlModRollCounter,VehicleSpeed_kmph,SteeringAngle_Deg,BatterySOC_Percent,BrakePedalStatus,AcceleratorPedalPos_Percent,RegenBrakingLevel,CoolantTemperature_C,AEB_Activation,LaneAssistanceStatus,CruiseControlSpeedSet_kmph,ParkingAssistEngaged,RainSensorStatus,TirePressure_FL_psi,TirePressure_FR_psi,TirePressure_RL_psi,TirePressure_RR_psi,LaneDepartureWarningActive,BatteryStateOfHealth_Percent,BatteryTemperature_Avg_C,BatteryMaxCellVoltage_V,BatteryMinCellVoltage_V,EvBatteryTemperature_C |           | index table:                            |
   |              |                  |                               |             |                |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | vehicle_db.dwd_vehicle_tbox_column_idx; |
   |              |                  |                               |             |                |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | task id:                                |
   |              |                  |                               |             |                |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | 8e60fb75-f01d-4a17-b1fc-9fb50ddc25a8;   |
   |              |                  |                               |             |                |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | partition by: [hash(128,vin)];          |
   |              |                  |                               |             |                |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | attributes: []                          |
   +--------------+------------------+-------------------------------+-------------+----------------+------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------+

   INDEX_STATE由BUILDING變?yōu)?code data-tag="code" code-type="xCode" class="code">ACTIVE時(shí)，表示索引已創(chuàng)建完成，此過程大概需等待3分鐘。

復(fù)雜查詢

1. （可選）車輛的電池健康數(shù)據(jù)按照一定時(shí)間頻率上傳至Lindorm，在執(zhí)行復(fù)雜查詢之前您可以先使用簡(jiǎn)單的查詢語句查看表中數(shù)據(jù)。

   例如，查詢某一時(shí)間段內(nèi)的10條電池?cái)?shù)據(jù)。

   SELECT  vin,BatteryStateOfHealth_Percent FROM  lindorm_columnar.vehicle_db.dwd_vehicle_tbox_column_idx WHERE sampletime >= 1686585600000 AND BatteryStateOfHealth_Percent <= 80 LIMIT 10;

   返回結(jié)果：

   單擊展開返回結(jié)果

   +----------------------------------+------------------------------+
   |               vin                | BatteryStateOfHealth_Percent |
   +----------------------------------+------------------------------+
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 23.764850669436843           |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 6.215661473439549            |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 12.852751497229441           |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 8.332735748118846            |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 19.596259310247877           |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 17.621117691572984           |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 66.77781508961095            |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 77.56923806011324            |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 25.969897299920774           |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 | 71.45633174979432            |
   +----------------------------------+------------------------------+

2. 查詢每輛車最后一次上報(bào)的記錄，并根據(jù)電池健康程度排序，查詢電池健康度最低的前10輛汽車和電池信息。

   SELECT vin,BatteryStateOfHealth_Percent FROM (SELECT *,ROW_NUMBER() OVER (PARTITION BY vin ORDER BY sampletime DESC) AS rn FROM lindorm_table.`default`.dwd_vehicle_tbox WHERE sampletime >= 1686585600000) AS tmp WHERE rn=1 ORDER BY BatteryStateOfHealth_Percent ASC LIMIT 10;

   返回結(jié)果：

   單擊展開返回結(jié)果

   +----------------------------------+------------------------------+
   |               vin                | BatteryStateOfHealth_Percent |
   +----------------------------------+------------------------------+
   | j878HLSbLco8jSVVyTY8CMUZqfkUs28v | 20.652739801362372           |
   | echVK3CaoAYBSPoKxbaCrMEW8y4BTpmg | 21.518064804363355           |
   | SLJ1mkIbwBaiSLZO9xc1gy0cbUK69ULc | 23.388871500485052           |
   | DgBXmx0idrLeSVq08RobnYPxl45sYchb | 27.994995362735676           |
   | 4pJpua6dn1jzTevlI0NsruBe91rVVix9 | 30.321789199354043           |
   | CgO3PchAjoKprtBrFbyCG5kK3UGkBcpQ | 31.9348912927222             |
   | 3gP6BebgBxACvguf0de6juIiwdoCWvWB | 33.72344917241882            |
   | iMJY8rDr08tFe4BHFQtkalhQ5KSdyOMD | 36.89775232730048            |
   | Dsn9Wzr65FVZWaq1ajeV6uH7rFLYOH8h | 44.50827494756204            |
   | Yxoi4mTxvWJB7hKmlPkCnbgK9nUY7Ba2 | 66.27943854842643            |
   +----------------------------------+------------------------------+

未創(chuàng)建搜索索引

基于電池健康和胎壓兩個(gè)維度，查詢過去一年中電池健康程度低（小于50）并且胎壓過低（胎壓指標(biāo)小于2） 的車輛。

-- 1686585600000 = 2023-06-13 0:0:00
SELECT DISTINCT(vin) FROM dwd_vehicle_tbox WHERE  sampletime >= 1686585600000 AND BatteryStateOfHealth_Percent < 50 AND (TirePressure_FL_psi < 2 OR TirePressure_FR_psi < 2 OR TirePressure_RL_psi < 2 OR TirePressure_RR_psi < 2);

返回結(jié)果：

+----------------------------------+
|               vin                |
+----------------------------------+
| 0JImIThOLvmYbKmhCP1Mact83E7XvF6s |
| 4spNfR8OhEsDXXaZADS3jivPQUGPXUQo |
| 59ydPuK3xdCvBV8wB1Y5UHsmWfO2aNQj |
| 5Kq9BkgTzNSexZ3fd3XL7AaF56NNVXHu |
| EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 |
| EwB2929l43ttvFNudMZDogppzbXfzYJ1 |
| FaXdxPwK0An7LbaeUV0r8PmqPoBz7Tn9 |
| KDx9AF9qrr4zhJR3sXsuEmVIANjTj8wg |
| LsvywOyn1lLjbrzEghQlHk3aXHfVr8Py |
| SivBFi3uzuUQT9bjO4jrIO7NEd9PYd55 |
| YC6zOBs7h57aqjIGZddDtzZ2YXfREnG2 |
| hkCS5GQ5sHBy2uyPEj7ifGXs21L4EUHu |
| k0kXqJMvx4gyOWfFUIHG6d4N2K8GFdDP |
| mHFfXFIGeItuZM4G9WYyqrEwyMrumsrZ |
| mXtpBVucRZVdKC2FH9MfP0i3MJCOsXXn |
| nH5YV2MAh3EDA14r2efpIHtIMp3AdsH4 |
| rXVDXz7JZ0PpvBgWu79bsKBWmcykxbnJ |
| rfsvhAgRbikInQ6O7gHDztLNWwQnZjVl |
| xU6qTtaDesFrSlQVLkZDe0vQE15gd5VE |
| yWuGjDH9fyRNAlF7k7HUe4GQFHHzSh9Y |
+----------------------------------+
20 rows in set (94912 ms)

該查詢?yōu)?b>全表掃描，總耗時(shí)為94912 ms。

使用搜索索引

1. 再次通過Lindorm-cli連接寬表引擎。

   ./lindorm-cli -url <jdbc url> -username <username> -password <password>

2. 創(chuàng)建搜索索引。

   CREATE INDEX IF NOT EXISTS dwd_vehicle_tbox_search_idx USING SEARCH ON dwd_vehicle_tbox (BatteryStateOfHealth_Percent,TirePressure_FL_psi,TirePressure_FR_psi,TirePressure_RL_psi,TirePressure_RR_psi);

3. （可選）查詢搜索索引的狀態(tài)。

   SHOW INDEX FROM dwd_vehicle_tbox;

   返回結(jié)果：

   +--------------+------------------+-------------------------------+-------------+-------------------------------+------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------+
   | TABLE_SCHEMA |    DATA_TABLE    |          INDEX_NAME           | INDEX_STATE |        INDEX_PROGRESS         | INDEX_TYPE | INDEX_COVERED |                                                                                                                                                                                                                                                                                         INDEX_COLUMN                                                                                                                                                                                                                                                                                          | INDEX_TTL |            INDEX_DESCRIPTION            |
   +--------------+------------------+-------------------------------+-------------+-------------------------------+------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------+
   | default      | dwd_vehicle_tbox | dwd_vehicle_tbox_search_idx   | ACTIVE      | N/A                           | SEARCH     | NA            | BatteryStateOfHealth_Percent,TirePressure_FL_psi,TirePressure_FR_psi,TirePressure_RL_psi,TirePressure_RR_psi                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | -1        |                                         |
   | default      | dwd_vehicle_tbox | column_index_dwd_vehicle_tbox | ACTIVE      | 2024-07-08 10:40:09.388 +0800 | COLUMNAR   | NA            | vin,sampletime,DriverStateLinkStatus,AirBagSystemStatus,HODHandoffmonitorValid,BodyCtrlModRollCounter,VehicleSpeed_kmph,SteeringAngle_Deg,BatterySOC_Percent,BrakePedalStatus,AcceleratorPedalPos_Percent,RegenBrakingLevel,CoolantTemperature_C,AEB_Activation,LaneAssistanceStatus,CruiseControlSpeedSet_kmph,ParkingAssistEngaged,RainSensorStatus,TirePressure_FL_psi,TirePressure_FR_psi,TirePressure_RL_psi,TirePressure_RR_psi,LaneDepartureWarningActive,BatteryStateOfHealth_Percent,BatteryTemperature_Avg_C,BatteryMaxCellVoltage_V,BatteryMinCellVoltage_V,EvBatteryTemperature_C |           | index table:                            |
   |              |                  |                               |             |                               |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | vehicle_db.dwd_vehicle_tbox_column_idx; |
   |              |                  |                               |             |                               |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | task id:                                |
   |              |                  |                               |             |                               |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | 8e60fb75-f01d-4a17-b1fc-9fb50ddc25a8;   |
   |              |                  |                               |             |                               |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | partition by: [hash(128,vin)];          |
   |              |                  |                               |             |                               |            |               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |           | attributes: []                          |
   +--------------+------------------+-------------------------------+-------------+-------------------------------+------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------+

   INDEX_STATE由BUILDING變?yōu)?code data-tag="code" code-type="xCode" class="code">ACTIVE時(shí)，表示索引已創(chuàng)建完成。

   說明

   如果索引狀態(tài)一直沒有變?yōu)锳CTIVE狀態(tài)，可以通過LTS控制臺(tái)查看搜索索引實(shí)時(shí)同步以及全量同步的執(zhí)行情況。

   查看方式：在LTS控制臺(tái)左側(cè)導(dǎo)航欄選擇Lindorm Search > 全量同步，單擊目標(biāo)任務(wù)名。如何登錄LTS，請(qǐng)參見登錄LTS。

4. 使用相同查詢語句查詢數(shù)據(jù)。

   基于電池健康和胎壓兩個(gè)維度，查詢過去一年電池健康程度低（小于50）且胎壓過低（胎壓指標(biāo)小于2） 的車輛。

   -- 1686585600000 = 2023-06-13 0:0:00
   SELECT DISTINCT(vin) FROM dwd_vehicle_tbox WHERE  sampletime >= 1686585600000 AND BatteryStateOfHealth_Percent < 50 AND (TirePressure_FL_psi < 2 OR TirePressure_FR_psi < 2 OR TirePressure_RL_psi < 2 OR TirePressure_RR_psi < 2);

   返回結(jié)果：

   單擊展開返回結(jié)果

   +----------------------------------+
   |               vin                |
   +----------------------------------+
   | 0JImIThOLvmYbKmhCP1Mact83E7XvF6s |
   | 4spNfR8OhEsDXXaZADS3jivPQUGPXUQo |
   | 59ydPuK3xdCvBV8wB1Y5UHsmWfO2aNQj |
   | 5Kq9BkgTzNSexZ3fd3XL7AaF56NNVXHu |
   | EEL5dgXYFjra8jF3tNOUuZPFL0Imh906 |
   | EwB2929l43ttvFNudMZDogppzbXfzYJ1 |
   | FaXdxPwK0An7LbaeUV0r8PmqPoBz7Tn9 |
   | KDx9AF9qrr4zhJR3sXsuEmVIANjTj8wg |
   | LsvywOyn1lLjbrzEghQlHk3aXHfVr8Py |
   | SivBFi3uzuUQT9bjO4jrIO7NEd9PYd55 |
   | YC6zOBs7h57aqjIGZddDtzZ2YXfREnG2 |
   | hkCS5GQ5sHBy2uyPEj7ifGXs21L4EUHu |
   | k0kXqJMvx4gyOWfFUIHG6d4N2K8GFdDP |
   | mHFfXFIGeItuZM4G9WYyqrEwyMrumsrZ |
   | mXtpBVucRZVdKC2FH9MfP0i3MJCOsXXn |
   | nH5YV2MAh3EDA14r2efpIHtIMp3AdsH4 |
   | rXVDXz7JZ0PpvBgWu79bsKBWmcykxbnJ |
   | rfsvhAgRbikInQ6O7gHDztLNWwQnZjVl |
   | xU6qTtaDesFrSlQVLkZDe0vQE15gd5VE |
   | yWuGjDH9fyRNAlF7k7HUe4GQFHHzSh9Y |
   +----------------------------------+
   20 rows in set (14321 ms)

   可以從返回結(jié)果看出，在相同查詢條件下未創(chuàng)建搜索索引時(shí)全表查詢耗時(shí)94912 ms，使用了搜索索引后查詢耗時(shí)14321 ms，時(shí)延大幅降低。

   說明

   搜索引擎在復(fù)雜多條件的查詢下延遲更低，更適用于涉及在線查詢業(yè)務(wù)較多的場(chǎng)景，而計(jì)算引擎更適合做更加復(fù)雜的查詢，例如有離線分析需求的場(chǎng)景。