分析列存索引查詢性能
PolarDB MySQL版新增列存索引查詢性能分析功能。該功能通過(guò)采集執(zhí)行SQL語(yǔ)句時(shí)的耗時(shí)等信息,并結(jié)合EXPLAIN獲得的查詢計(jì)劃一并返回給用戶。能夠幫助用戶了解執(zhí)行SQL語(yǔ)句時(shí)的耗時(shí)細(xì)節(jié),并且能夠輔助分析慢SQL。
版本要求
集群版本需為PolarDB MySQL版8.0.1.1.42及以上,您可以通過(guò)查詢版本號(hào)來(lái)確認(rèn)集群版本。
使用說(shuō)明
您需要將參數(shù)imci_analyze_query
的值設(shè)置為ON,來(lái)開(kāi)啟列存索引查詢性能分析功能。
參數(shù)名稱 | 級(jí)別 | 說(shuō)明 |
imci_analyze_query | Session | 列存索引查詢性能分析功能控制開(kāi)關(guān)。取值范圍如下:
|
示例
簡(jiǎn)單查詢示例
以在TPC-H Schema上執(zhí)行的一個(gè)簡(jiǎn)單查詢語(yǔ)句為例,SQL語(yǔ)句如下:
SELECT l_shipmode, COUNT(*) FROM lineitem, orders WHERE l_orderkey = o_orderkey GROUP BY l_shipmode;
首先,您可以通過(guò)EXPLAIN得到該查詢優(yōu)化器估計(jì)的各個(gè)算子的執(zhí)行代價(jià)以及返回行數(shù)信息。
EXPLAIN SELECT l_shipmode, COUNT(*) FROM lineitem, orders WHERE l_orderkey = o_orderkey GROUP BY l_shipmode;
查詢結(jié)果如下:
+----+------------------------+----------+--------+----------+---------------------------------------------------------------+ | ID | Operator | Name | E-Rows | E-Cost | Extra Info | +----+------------------------+----------+--------+----------+---------------------------------------------------------------+ | 1 | Select Statement | | | | IMCI Execution Plan (max_dop = 1, max_query_mem = unlimited) | | 2 | └─Hash Groupby | | 6 | 51218.50 | Group Key: lineitem.l_shipmode | | 3 | └─Compute Scalar | | 1869 | 50191.00 | | | 4 | └─Hash Join | | 1869 | 31000.00 | Join Cond: lineitem.l_orderkey = orders.o_orderkey | | 5 | ├─Table Scan | lineitem | 2000 | 0.00 | | | 6 | └─Table Scan | orders | 2000 | 0.00 | | +----+------------------------+----------+--------+----------+---------------------------------------------------------------+
開(kāi)啟性能分析功能。
SET imci_analyze_query = ON;
執(zhí)行SQL語(yǔ)句。
SELECT l_shipmode, COUNT(*) FROM lineitem, orders WHERE l_orderkey = o_orderkey GROUP BY l_shipmode;
執(zhí)行結(jié)果如下:
+------------+----------+ | l_shipmode | COUNT(*) | +------------+----------+ | REG AIR | 283 | | SHIP | 269 | | FOB | 299 | | RAIL | 289 | | TRUCK | 314 | | MAIL | 274 | | AIR | 272 | +------------+----------+ 7 rows in set (0.05 sec)
性能分析結(jié)果會(huì)存放在
information_schema.imci_sql_profiling
表中。在將參數(shù)imci_analyze_query
的值設(shè)置為ON時(shí),該表將存放最近一條使用列存索引執(zhí)行的查詢語(yǔ)句的性能分析信息,通過(guò)查詢這張表,就可以獲得執(zhí)行的查詢語(yǔ)句的性能信息。查詢SQL語(yǔ)句的性能信息。
/*ROUTE_TO_LAST_USED*/SELECT * FROM information_schema.imci_sql_profiling;
說(shuō)明/*ROUTE_TO_LAST_USED*/
用于指示智能代理將查詢路由到上一條語(yǔ)句執(zhí)行的節(jié)點(diǎn)上。否則可能會(huì)因?yàn)槁酚刹坏讲樵儓?zhí)行的節(jié)點(diǎn),導(dǎo)致查詢不到結(jié)果或結(jié)果不正確。查詢結(jié)果如下:
+----+------------------------+----------+--------+----------+-------------------+--------------------------------------------------------------------------------------------------------+ | ID | Operator | Name | A-Rows | A-Cost | Execution Time(s) | Extra Info | +----+------------------------+----------+--------+----------+-------------------+--------------------------------------------------------------------------------------------------------+ | 1 | Select Statement | | 0 | 52609.51 | 0 | IMCI Execution Plan (max_dop = 1, real_dop = 1, max_query_mem = unlimited, real_query_mem = unlimited) | | 2 | └─Hash Groupby | | 7 | 52609.5 | 0.002 | Group Key: lineitem.l_shipmode | | 3 | └─Compute Scalar | | 2000 | 51501 | 0 | | | 4 | └─Hash Join | | 2000 | 31000 | 0.007 | Join Cond: lineitem.l_orderkey = orders.o_orderkey | | 5 | ├─Table Scan | lineitem | 2000 | 0 | 0.001 | | | 6 | └─Table Scan | orders | 2000 | 0 | 0 | | +----+------------------------+----------+--------+----------+-------------------+--------------------------------------------------------------------------------------------------------+
可以看到,
Hash Join
與Hash Groupby
對(duì)應(yīng)的Extra Info
信息中標(biāo)明了對(duì)應(yīng)的連接條件以及分組列,而第1行的Extra Info
則展示了執(zhí)行查詢操作時(shí)使用的CPU和內(nèi)存信息。(可選)您可以在
information_schema.imci_sql_profiling
表上執(zhí)行聚合等操作。如查看該查詢語(yǔ)句的執(zhí)行時(shí)間等。/*ROUTE_TO_LAST_USED*/SELECT SUM(`Execution Time(s)`) AS TOTAL_TIME FROM information_schema.imci_sql_profiling;
執(zhí)行結(jié)果如下:
+----------------------+ | TOTAL_TIME | +----------------------+ | 0.010000000000000002 | +----------------------+ 1 row in set (0.00 sec)
可以看到,該查詢語(yǔ)句的執(zhí)行耗時(shí)為10ms。
復(fù)雜查詢示例
以在TPC-H SF100數(shù)據(jù)集上執(zhí)行一個(gè)復(fù)雜查詢?yōu)槔ㄟ^(guò)EXPLAIN和查詢分析功能分析查詢耗時(shí)分布以及性能。SQL語(yǔ)句如下:
SELECT
c_name,
sum(l_quantity)
FROM
customer,
orders,
lineitem
WHERE
o_orderkey IN (
SELECT
l_orderkey
FROM
lineitem
WHERE
l_partkey > 18000000
)
AND c_custkey = o_custkey
AND o_orderkey = l_orderkey
GROUP BY
c_name
ORDER BY
c_name
LIMIT 10;
通過(guò)EXPLAIN查看該查詢語(yǔ)句的執(zhí)行計(jì)劃。
EXPLAIN SELECT c_name, sum(l_quantity) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem WHERE l_partkey > 18000000 ) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name ORDER BY c_name LIMIT 10;
執(zhí)行結(jié)果如下:
+----+----------------------------------+----------+-----------+------------+---------------------------------------------------------------+ | ID | Operator | Name | E-Rows | E-Cost | Extra Info | +----+----------------------------------+----------+-----------+------------+---------------------------------------------------------------+ | 1 | Select Statement | | | | IMCI Execution Plan (max_dop = 32, max_query_mem = unlimited) | | 2 | └─Limit | | 10 | 7935739.96 | Offset=0 Limit=10 | | 3 | └─Sort | | 10 | 7935739.96 | Sort Key: c_name ASC | | 4 | └─Hash Groupby | | 1503700 | 7933273.99 | Group Key: customer.C_NAME | | 5 | └─Hash Right Semi Join | | 54010545 | 7865930.26 | Join Cond: lineitem.L_ORDERKEY = orders.O_ORDERKEY | | 6 | ├─Table Scan | lineitem | 59785766 | 24001.52 | Cond: (L_PARTKEY > 18000000) | | 7 | └─Hash Join | | 538776190 | 5488090.33 | Join Cond: orders.O_ORDERKEY = lineitem.L_ORDERKEY | | 8 | ├─Hash Join | | 181006430 | 668535.99 | Join Cond: customer.C_CUSTKEY = orders.O_CUSTKEY | | 9 | │ ├─Table Scan | customer | 15000000 | 600.00 | | | 10 | │ └─Table Scan | orders | 150000000 | 6000.00 | | | 11 | └─Table Scan | lineitem | 600037902 | 24001.52 | | +----+----------------------------------+----------+-----------+------------+---------------------------------------------------------------+
該SQL語(yǔ)句的執(zhí)行代價(jià)為7935739。其中,ID為7的Hash Join會(huì)消耗大量CPU(其代價(jià)為5488090,占總代價(jià)約70%)。若要驗(yàn)證這條查詢語(yǔ)句的性能問(wèn)題,您可以開(kāi)啟性能分析功能,開(kāi)啟后再執(zhí)行該SQL語(yǔ)句,并通過(guò)查看SQL語(yǔ)句的查詢性能信息來(lái)分析具體原因。
開(kāi)啟性能分析功能。
SET imci_analyze_query = ON;
執(zhí)行SQL語(yǔ)句。
SELECT c_name, sum(l_quantity) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem WHERE l_partkey > 18000000 ) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name ORDER BY c_name LIMIT 10;
執(zhí)行結(jié)果如下:
+--------------------+-----------------+ | c_name | sum(l_quantity) | +--------------------+-----------------+ | Customer#000000001 | 172.00 | | Customer#000000002 | 663.00 | | Customer#000000004 | 174.00 | | Customer#000000005 | 488.00 | | Customer#000000007 | 1135.00 | | Customer#000000008 | 440.00 | | Customer#000000010 | 625.00 | | Customer#000000011 | 143.00 | | Customer#000000013 | 1032.00 | | Customer#000000014 | 564.00 | +--------------------+-----------------+ 10 rows in set (21.37 sec)
查詢SQL語(yǔ)句的性能信息。
/*ROUTE_TO_LAST_USED*/SELECT * FROM information_schema.imci_sql_profiling;
說(shuō)明/*ROUTE_TO_LAST_USED*/
用于指示智能代理將查詢路由到上一條語(yǔ)句執(zhí)行的節(jié)點(diǎn)上。否則可能會(huì)因?yàn)槁酚傻藉e(cuò)誤節(jié)點(diǎn)而查詢不到結(jié)果。查詢結(jié)果如下:
+----+----------------------------------+----------+-----------+------------+-------------------+----------------------------------------------------+ | ID | Operator | Name | A-Rows | A-Cost | Execution Time(s) | Extra Info | +----+----------------------------------+----------+-----------+------------+-------------------+----------------------------------------------------+ | 1 | Select Statement | | 0 | 8336856.67 | 0 | | | 2 | └─Limit | | 10 | 8336856.67 | 0.002 | Offset=0 Limit=10 | | 3 | └─Sort | | 0 | 8336856.67 | 2.275 | Sort Key: c_name ASC | | 4 | └─Hash Groupby | | 9813586 | 8320763.22 | 160.083 | Group Key: customer.C_NAME | | 5 | └─Hash Right Semi Join | | 239598134 | 7994854.23 | 98.174 | Join Cond: lineitem.L_ORDERKEY = orders.O_ORDERKEY | | 6 | ├─Table Scan | lineitem | 60013756 | 24001.52 | 3.28 | Cond: (L_PARTKEY > 18000000) | | 7 | └─Hash Join | | 600037902 | 5156677.35 | 301.503 | Join Cond: orders.O_ORDERKEY = lineitem.L_ORDERKEY | | 8 | ├─Hash Join | | 150000000 | 629777.96 | 97.201 | Join Cond: customer.C_CUSTKEY = orders.O_CUSTKEY | | 9 | │ ├─Table Scan | customer | 15000000 | 600 | 3.321 | | | 10 | │ └─Table Scan | orders | 150000000 | 6000 | 0.241 | | | 11 | └─Table Scan | lineitem | 600037902 | 24001.52 | 0.661 | | +----+----------------------------------+----------+-----------+------------+-------------------+----------------------------------------------------+
根據(jù)查詢的性能信息可以看出:Hash Join(7)耗費(fèi)了約一半的時(shí)間。這是因?yàn)?code data-tag="code" code-type="xCode" class="code">lineitem表與Hash Join(8)表的結(jié)果都很大,從而導(dǎo)致join的數(shù)據(jù)量很大,因此耗時(shí)很長(zhǎng)。在Hash Join中,構(gòu)建哈希表的開(kāi)銷通常遠(yuǎn)遠(yuǎn)大于哈希查找,因此只需要減小Join(8)的結(jié)果大小,就能有效減少查詢的耗時(shí)。
查看SQL語(yǔ)句和執(zhí)行計(jì)劃,若優(yōu)化器將
o_orderkey in (...)
轉(zhuǎn)換為半連接子查詢,且能夠?qū)⒃摋l件下推,則能提高該SQL語(yǔ)句的查詢性能。開(kāi)啟基于代價(jià)的半連接下推功能。
SET imci_optimizer_switch = 'semijoin_pushdown=on';
通過(guò)EXPLAIN查看該查詢語(yǔ)句的執(zhí)行計(jì)劃。
EXPLAIN SELECT c_name, sum(l_quantity) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem WHERE l_partkey > 18000000 ) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name ORDER BY c_name LIMIT 10;
執(zhí)行結(jié)果如下:
+----+----------------------------------------+----------+-----------+------------+---------------------------------------------------------------+ | ID | Operator | Name | E-Rows | E-Cost | Extra Info | +----+----------------------------------------+----------+-----------+------------+---------------------------------------------------------------+ | 1 | Select Statement | | | | IMCI Execution Plan (max_dop = 32, max_query_mem = unlimited) | | 2 | └─Limit | | 10 | 2800433.74 | Offset=0 Limit=10 | | 3 | └─Sort | | 10 | 2800433.74 | Sort Key: c_name ASC | | 4 | └─Hash Groupby | | 14567321 | 2776544.58 | Group Key: customer.C_NAME | | 5 | └─Hash Join | | 57918330 | 2631846.75 | Join Cond: orders.O_ORDERKEY = lineitem.L_ORDERKEY | | 6 | ├─Hash Join | | 14567321 | 1014041.92 | Join Cond: orders.O_CUSTKEY = customer.C_CUSTKEY | | 7 | │ ├─Hash Right Semi Join | | 12071937 | 906226.67 | Join Cond: lineitem.L_ORDERKEY = orders.O_ORDERKEY | | 8 | │ │ ├─Table Scan | lineitem | 59785766 | 24001.52 | Cond: (L_PARTKEY > 18000000) | | 9 | │ │ └─Table Scan | orders | 150000000 | 6000.00 | | | 10 | │ └─Table Scan | customer | 15000000 | 600.00 | | | 11 | └─Table Scan | lineitem | 600037902 | 24001.52 | | +----+----------------------------------------+----------+-----------+------------+---------------------------------------------------------------+
可以看出,開(kāi)啟半連接下推功能后,查詢總代價(jià)從7935739下降至2800433。
重新執(zhí)行該SQL語(yǔ)句。
SELECT c_name, sum(l_quantity) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem WHERE l_partkey > 18000000 ) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name ORDER BY c_name LIMIT 10;
執(zhí)行結(jié)果如下:
+--------------------+-----------------+ | c_name | sum(l_quantity) | +--------------------+-----------------+ | Customer#000000001 | 172.00 | | Customer#000000002 | 663.00 | | Customer#000000004 | 174.00 | | Customer#000000005 | 488.00 | | Customer#000000007 | 1135.00 | | Customer#000000008 | 440.00 | | Customer#000000010 | 625.00 | | Customer#000000011 | 143.00 | | Customer#000000013 | 1032.00 | | Customer#000000014 | 564.00 | +--------------------+-----------------+ 10 rows in set (13.74 sec)
可以看出,執(zhí)行耗時(shí)降低了約40%。
查詢SQL語(yǔ)句的性能信息。
/*ROUTE_TO_LAST_USED*/SELECT * FROM information_schema.imci_sql_profiling;
執(zhí)行結(jié)果如下:
+----+----------------------------------------+----------+-----------+------------+-------------------+----------------------------------------------------+ | ID | Operator | Name | A-Rows | A-Cost | Execution Time(s) | Extra Info | +----+----------------------------------------+----------+-----------+------------+-------------------+----------------------------------------------------+ | 1 | Select Statement | | 0 | 4318488.35 | 0 | | | 2 | └─Limit | | 10 | 4318488.35 | 0.002 | Offset=0 Limit=10 | | 3 | └─Sort | | 0 | 4318488.34 | 3.076 | Sort Key: c_name ASC | | 4 | └─Hash Groupby | | 9813586 | 4302394.89 | 163.149 | Group Key: customer.C_NAME | | 5 | └─Hash Join | | 239598134 | 3976485.91 | 151.253 | Join Cond: orders.O_ORDERKEY = lineitem.L_ORDERKEY | | 6 | ├─Hash Join | | 49393149 | 1321335.54 | 55.392 | Join Cond: orders.O_CUSTKEY = customer.C_CUSTKEY | | 7 | │ ├─Hash Right Semi Join | | 49393149 | 954805.16 | 52.552 | Join Cond: lineitem.L_ORDERKEY = orders.O_ORDERKEY | | 8 | │ │ ├─Table Scan | lineitem | 60013756 | 24001.52 | 2.791 | Cond: (L_PARTKEY > 18000000) | | 9 | │ │ └─Table Scan | orders | 150000000 | 6000 | 0.152 | | | 10 | │ └─Table Scan | customer | 15000000 | 600 | 0.028 | | | 11 | └─Table Scan | lineitem | 600037902 | 24001.52 | 0.642 | | +----+----------------------------------------+----------+-----------+------------+-------------------+----------------------------------------------------+
從性能分析結(jié)果可以看出,半連接被下推,之前的大表Join被消除,查詢耗時(shí)顯著減少。