首頁
云原生數(shù)據(jù)庫 PolarDB
云原生數(shù)據(jù)庫PolarDB分布式版
用戶指南
數(shù)據(jù)導(dǎo)入和導(dǎo)出
使用程序?qū)霐?shù)據(jù)
使用程序?qū)霐?shù)據(jù)
更新時(shí)間:
本文將介紹如何通過編寫代碼的方式,將數(shù)據(jù)導(dǎo)入到PolarDB-X中。
以如下表結(jié)構(gòu)示例進(jìn)行說明:
CREATE TABLE `sourcedb` (
`id` int(11) NOT NULL,
`k` int(11) NOT NULL DEFAULT '0',
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `k_1` (`k`)
) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4 dbpartition by hash(`id`);
從數(shù)據(jù)庫中導(dǎo)出源數(shù)據(jù)
源數(shù)據(jù)可以由用戶自行生成,也可以從數(shù)據(jù)庫中導(dǎo)出,本文以mysql -e
命令的方式導(dǎo)出進(jìn)行示例,具體方法如下:
mysql -h ip -P port -u usr -pPassword db_name -N -e "SELECT id,k,c,pad FROM sourcedb;" >/home/data_1000w.txt
## 原始數(shù)據(jù)以制表符分隔,數(shù)據(jù)格式:188092293 27267211 59775766593-64673028018-...-09474402685 01705051424-...-54211554755
mysql -h ip -P port -u usr -pPassword db_name -N -e "SELECT id,k,c,pad FROM sourcedb;" | sed 's/\t/,/g' >/home/data_1000w.csv
## csv文件格式以逗號(hào)分隔,數(shù)據(jù)格式:188092293,27267211,59775766593-64673028018-...-09474402685,01705051424-...-54211554755
推薦將字符串進(jìn)行處理,轉(zhuǎn)變成csv文件格式,方便后續(xù)程序讀取數(shù)據(jù)。
在PolarDB-X中創(chuàng)建目標(biāo)表
源數(shù)據(jù)不包括建表語句,需要手動(dòng)在PolarDB-X目標(biāo)數(shù)據(jù)庫上創(chuàng)建表,示例如下:
CREATE TABLE `targetdb` (
`id` int(11) NOT NULL,
`k` int(11) NOT NULL DEFAULT '0',
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `k_1` (`k`)
) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4 dbpartition by hash(`id`);
使用程序?qū)霐?shù)據(jù)到PolarDB-X
您可以自行編寫程序,連接PolarDB-X,然后讀取本地?cái)?shù)據(jù),通過Batch Insert
語句導(dǎo)入到PolarDB-X中。
下面是一個(gè)簡(jiǎn)單的Java程序示例:
// 需要mysql-connector-java.jar, 詳情界面:https://mvnrepository.com/artifact/mysql/mysql-connector-java
// 下載鏈接:https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.20/mysql-connector-java-8.0.20.jar
// 注:不同版本的mysql-connector-java.jar,可能Class.forName("com.mysql.cj.jdbc.Driver")類路徑不同
// 編譯 javac LoadData.java
// 運(yùn)行 java -cp .:mysql-connector-java-8.0.20.jar LoadData
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
public class LoadData {
public static void main(String[] args) throws IOException, ClassNotFoundException, SQLException {
File dataFile = new File("/home/data_1000w.csv");
String sql = "insert into targetdb(id, k, c, pad) values(?, ?, ?, ?)";
int batchSize = 1000;
try (
Connection connection = getConnection("ip", 3306, "db", "usr", "password");
BufferedReader br = new BufferedReader(new FileReader(dataFile))) {
String line;
PreparedStatement st = connection.prepareStatement(sql);
long startTime = System.currentTimeMillis();
int batchCount = 0;
while ((line = br.readLine()) != null) {
String[] data = line.split(",");
st.setInt(1, Integer.valueOf(data[0]));
st.setInt(2, Integer.valueOf(data[1]));
st.setObject(3, data[2]);
st.setObject(4, data[3]);
st.addBatch();
if (++batchCount % batchSize == 0) {
st.executeBatch();
System.out.println(String.format("insert %d records", batchCount));
}
}
if (batchCount % batchSize != 0) {
st.executeBatch();
}
long cost = System.currentTimeMillis() - startTime;
System.out.println(String.format("Take %d second,insert %d records, tps %d", cost/1000, batchCount, batchCount/(cost/1000)));
}
}
/**
* 獲取數(shù)據(jù)庫連接
*
* @param host 數(shù)據(jù)庫地址
* @param port 端口
* @param database 數(shù)據(jù)庫名稱
* @param username 用戶名
* @param password 密碼
* @return
* @throws ClassNotFoundException
* @throws SQLException
*/
private static Connection getConnection(String host, int port, String database, String username, String password)
throws ClassNotFoundException, SQLException {
Class.forName("com.mysql.cj.jdbc.Driver");
String url = String.format(
"jdbc:mysql://%s:%d/%s?autoReconnect=true&socketTimeout=600000&rewriteBatchedStatements=true", host, port,
database);
Connection con = DriverManager.getConnection(url, username, password);
return con;
}
}
您可以根據(jù)實(shí)際應(yīng)用場(chǎng)景編寫程序,設(shè)置合適的batch size和多線程導(dǎo)入,來提高導(dǎo)入的性能。
文檔內(nèi)容是否對(duì)您有幫助?