/` |
diff --git a/docs/assets/export-wizard.png b/docs/assets/export-wizard.png
index e21036b..b05e75a 100644
Binary files a/docs/assets/export-wizard.png and b/docs/assets/export-wizard.png differ
diff --git a/docs/assets/import-package.png b/docs/assets/import-package.png
index 906bdfd..a29e796 100644
Binary files a/docs/assets/import-package.png and b/docs/assets/import-package.png differ
diff --git a/docs/assets/import-parquet.png b/docs/assets/import-parquet.png
new file mode 100644
index 0000000..476e159
Binary files /dev/null and b/docs/assets/import-parquet.png differ
diff --git a/docs/assets/login.png b/docs/assets/login.png
index 0ca255c..96f635a 100644
Binary files a/docs/assets/login.png and b/docs/assets/login.png differ
diff --git a/docs/assets/main-dark.png b/docs/assets/main-dark.png
index 72710d3..c2ef415 100644
Binary files a/docs/assets/main-dark.png and b/docs/assets/main-dark.png differ
diff --git a/docs/assets/main-light.png b/docs/assets/main-light.png
index e00ee57..9a3fc8d 100644
Binary files a/docs/assets/main-light.png and b/docs/assets/main-light.png differ
diff --git a/docs/assets/main-nord.png b/docs/assets/main-nord.png
index 9525589..418e226 100644
Binary files a/docs/assets/main-nord.png and b/docs/assets/main-nord.png differ
diff --git a/docs/assets/package-viewer.png b/docs/assets/package-viewer.png
new file mode 100644
index 0000000..f262b2b
Binary files /dev/null and b/docs/assets/package-viewer.png differ
diff --git a/docs/assets/sqljam-tutorial.mp4 b/docs/assets/sqljam-tutorial.mp4
new file mode 100644
index 0000000..8c5bdb0
Binary files /dev/null and b/docs/assets/sqljam-tutorial.mp4 differ
diff --git a/docs/assets/table-data.png b/docs/assets/table-data.png
index 17da667..292df00 100644
Binary files a/docs/assets/table-data.png and b/docs/assets/table-data.png differ
diff --git a/docs/blogger/sqljam-en.md b/docs/blogger/sqljam-en.md
index 50d7e65..28daa11 100644
--- a/docs/blogger/sqljam-en.md
+++ b/docs/blogger/sqljam-en.md
@@ -6,8 +6,8 @@
**SqlJam**: *Move tables across databases. Schema, data and all.*
-A Java 17 + JavaFX desktop tool that copies tables between **MySQL, PostgreSQL, Oracle, SQL Server, H2, SQLite** and more, in any direction:
-**import directly** into a target database, or save a self-describing **SQL export package** and import it later.
+A Java 17 + JavaFX desktop tool that copies tables between relational and OLAP databases (**MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, H2, SQLite, DuckDB, ClickHouse** and more) in any direction:
+**import directly** into a target database or into the same schema as copies, or save a self-describing **export package** of SQL or **Parquet** files and import it later.
@@ -25,6 +25,11 @@ SqlJam reads **one metadata tree** and generates SQL with a **dialect chosen by
| Huge scripts | Data files split at 10 MB: `orders.sql`, `orders_2.sql` … |
| LOBs | Stored as files, restored by primary key |
| Scripts without context | `manifest.json`: source, target, options, SHA-256 per file, row counts |
+| Analytics needs Parquet | Parquet packages through an embedded DuckDB, Parquet files of other tools loaded into any table |
+| Warehouses apart from OLTP | DuckDB and ClickHouse data sources, grouped as OLAP databases |
+| Only part of a table | Data viewer query (columns, WHERE, GROUP BY, ORDER BY) and export of the rows of the query |
+| A copy next to the original | Copies in the same schema with `{table}_copy`, keys and indexes renamed, source tables protected |
+| Custom and modern types | PostgreSQL enums and domains, vectors, unions, maps, tuples and variants |
## 3. Quick Start
@@ -43,7 +48,8 @@ bin/
├── sqljam.sh / sqljam.bat # launchers, pick the jar of the platform
├── sqljam.properties # configuration, edit and restart
├── sqljam.vmoptions # JVM options, one per line
-└── sqljam.png # Dock icon of macOS
+├── sqljam.png # Dock icon of macOS
+└── sqljam-tutorial.mp4 # tutorial video, export and import in under 3 minutes
```
`sqljam.properties` and `sqljam.vmoptions` sit next to the runnable jar. Edit them and restart, no rebuild needed.
@@ -52,10 +58,20 @@ A splash screen shows the loading of the configuration, the data sources and the
+The first start brings an **Example Shop (H2)** data source with a few e-commerce tables (categories, customers, products, orders, order_items) to try everything right away. **Example** on the login page or **Help → Restore Example Database** brings it back at any time.
+
+Right-click a data source to **Connect**, and once it is connected the same menu offers **Disconnect**, which also closes its table tabs. When an import or export needs a target data source, the **+** button next to it creates one on the spot. **Exit** sits at the right end of the toolbar. **Help → About SqlJam** shows the version, the home page, the repository and how to reach the author, and the GitHub icon at the bottom right opens the repository.
+
+A short [tutorial video](https://github.com/paganini2008/sqljam/blob/master/docs/assets/sqljam-tutorial.mp4) walks through the example database: query rows, export into files, import the package into PostgreSQL, and copy the tables straight into SQL Server and Oracle.
+
| 1. Log in to a data source | 2. Export wizard | 3. Progress |
|---|---|---|
|  |  |  |
+| 4. Data query | 5. Package viewer | 6. Parquet files |
+|---|---|---|
+|  |  |  |
+
## 4. Requirements
Only **Java 17+** is needed to run SqlJam. JavaFX and the JDBC drivers are inside the runnable jar, and building needs no Maven installation thanks to the Maven Wrapper (`./mvnw`).
@@ -73,7 +89,10 @@ Only **Java 17+** is needed to run SqlJam. JavaFX and the JDBC drivers are insid
| PostgreSQL | 9.x to 16 |
| Oracle | 11g to 23ai |
| SQL Server | 2008 to 2022 |
+| MariaDB | 10.3 to 11.x |
| H2 / SQLite | 2.x / 3.x |
+| DuckDB (OLAP) | 1.x |
+| ClickHouse (OLAP) | 24.x to 26.x |
All JDBC drivers are bundled in the fat jar.
@@ -88,11 +107,14 @@ flowchart LR
D --> I[Import directly] --> TD[(Target DB)]
D --> E[Export package
+首次启动会自带一个 **Example Shop (H2)** 示例数据源，包含几张电商表（categories、customers、products、orders、order_items），装好就能直接体验。删掉了也没关系，登录页的 **Example** 按钮或菜单 **Help → Restore Example Database** 随时可以恢复。
+
+右键数据源选 **Connect**，连上之后同一个菜单变成 **Disconnect**，断开时它的表页签一起关闭。导入导出选目标数据源时，旁边的 **+** 按钮可以当场新建一个。工具栏最右边是 **Exit**。**Help → About SqlJam** 里有版本、主页、仓库地址和作者联系方式，右下角的 GitHub 图标点一下直接打开仓库。
+
+[教学视频](https://github.com/paganini2008/sqljam/blob/master/docs/assets/sqljam-tutorial.mp4)以示例库演示一遍：筛选行、导出到文件、导出包导入 PostgreSQL，再直接复制到 SQL Server 和 Oracle。
+
| 1. 登录数据源 | 2. 导出向导 | 3. 进度 |
|---|---|---|
|  |  |  |
+| 4. 数据查询 | 5. 导出包浏览 | 6. Parquet 文件 |
+|---|---|---|
+|  |  |  |
+
## 4. Requirements
运行只需要 **Java 17+**。JavaFX 和 JDBC 驱动都打进了可运行 jar，从源码构建也不需要安装 Maven，项目自带 Maven Wrapper（`./mvnw`）。
@@ -73,7 +89,10 @@ bin/
| PostgreSQL | 9.x ~ 16 |
| Oracle | 11g ~ 23ai |
| SQL Server | 2008 ~ 2022 |
+| MariaDB | 10.3 ~ 11.x |
| H2 / SQLite | 2.x / 3.x |
+| DuckDB（OLAP） | 1.x |
+| ClickHouse（OLAP） | 24.x ~ 26.x |
JDBC 驱动全部打进 fat jar，无需额外下载。
@@ -88,11 +107,14 @@ flowchart LR
D --> I[直接导入] --> TD[(目标库)]
D --> E[导出包SqlJam · Move tables across databases
-
SqlJam Move tables across databases. Schema, data and all.
-
A desktop import/export tool that copies tables, indexes, constraints, sequences, partitions and rows between databases, directly or through a self-describing SQL export package you can import later.
+
A desktop import/export tool that copies tables, indexes, constraints, sequences, partitions and rows between relational and OLAP databases, directly, as copies in the same schema, or through a self-describing SQL or Parquet export package you can import later.
- MySQL PostgreSQL Oracle
+ MySQL MariaDB PostgreSQL Oracle
SQL Server H2 SQLite
+ DuckDB ClickHouse
@@ -229,6 +231,10 @@ Everything a table needs on the other side
Verified imports Status, target type and SHA-256 checked before running.
Progress you can see Overall percentage, per-table progress, error log and cancel.
Dark by default A modern JavaFX UI with 7 switchable themes.
Parquet packages Columnar export packages and Parquet files of other tools, through an embedded DuckDB.
OLAP sources DuckDB and ClickHouse next to the relational databases.
Query & export Columns, WHERE, GROUP BY and ORDER BY in the data viewer, then export just those rows.
Copies in place Copy tables inside the same schema with names like {table}_copy.
Read once, generate for any target
1 Metadata : per-database operations read tables, keys, indexes, comments, partitions and sequences.
2 Dialect : DbType.createDialect(major, minor) picks e.g. Oracle11gDialect.
-
3 Rows : counted, paged by primary key, normalized and written in batches.
+
3 Rows : counted, paged by primary key, normalized and written in batches, or written to Parquet by an embedded DuckDB.
@@ -289,12 +295,13 @@ Result
Use cases
-
Three ways to move a table
+
Ways to move a table
Export a package
Import directly
Import a package
- Browse & preview
+ Query & export
+ Parquet
@@ -304,7 +311,7 @@
Three ways to move a table
-
Copy straight into another data source. Target schemas are created, identities continue from the imported max value.
+
Copy straight into another data source, or create one on the spot with the + button next to it. Target schemas are created, identities continue from the imported max value.
target.setDbType(DbType.SQLSERVER);
target.setTargetSchemaName("hr" ); // created if missing
importer.exportDdlAndData();
@@ -316,8 +323,18 @@
Three ways to move a table
importer.importDirectory(
new File(
"export" ));
-
-
Browse databases, schemas and tables. Inspect columns, indexes, foreign keys and DDL in any dialect, then page through rows.
+
+
Browse databases, schemas and tables. Narrow the rows with columns, WHERE, GROUP BY and ORDER BY, then export just those rows. A missing target table is created with the selected columns.
+
source.getTableQueries().put("EMPLOYEES" , new TableQuery(columns,
+ "SALARY > 1000" , null , "SALARY DESC" ));
+target.setTableNamePattern("{table}_copy" ); // copies in the same schema
+
+
+
Export packages of Parquet files, look inside them with sizes, times and previews, and load Parquet files of other tools into any table.
+
parquet.setCompression("ZSTD" );
+parquet.exportDdlAndData(); // data/<table>.parquet
+importer.importFiles(files, "sales" , ParquetImporter.Mode.CREATE);
@@ -330,7 +347,7 @@ Tens of thousands of rows per second
95.3k rows/s · H2 → Oracle
100% cross-database pairs pass
-
303 automated tests
+
578 automated tests
91% line coverage
@@ -360,7 +377,7 @@ Learn more
Getting Started Build, run and export your first tables.
Read → Examples Export packages, direct imports and type translation.
Read → Configuration Every key of sqljam.properties and export options.
Read → API Reference Exporter, ScriptExporter, ImportExporter, ScriptImporter.
Browse → API Reference Exporter, ScriptExporter, ImportExporter, ScriptImporter, ParquetExporter, ParquetImporter.
Browse → Architecture Metadata tree, dialects and version subclasses.
Read → Blog English, 中文 and a Medium edition.
Read → 17.0.13
2.0.1
12.3.1
+ 1.5.6.0
+ 0.10.0
1.18.34
com.github.sqljam.face.Launcher
@@ -65,7 +67,7 @@
commons-io
commons-io
- 2.17.0
+ 2.20.0
com.zaxxer
@@ -112,6 +114,11 @@
+
+ org.mariadb.jdbc
+ mariadb-java-client
+ 3.5.10
+
org.postgresql
postgresql
@@ -142,6 +149,17 @@
sqlite-jdbc
3.46.1.3
+
+
+ org.duckdb
+ duckdb_jdbc
+ ${duckdb.version}
+
+
+ com.clickhouse
+ clickhouse-jdbc
+ ${clickhouse.version}
+
@@ -553,6 +571,13 @@
*.png
+
+
+ ${project.basedir}/docs/assets
+
+ sqljam-tutorial.mp4
+
+
@@ -593,4 +618,26 @@
+
+
+
+ ui
+
+
+
+ org.apache.maven.plugins
+ maven-surefire-plugin
+
+
+ glass
+ false
+ 30000
+ en
+
+
+
+
+
+
+
diff --git a/src/assembly/linux.xml b/src/assembly/linux.xml
index f98971c..2db831a 100644
--- a/src/assembly/linux.xml
+++ b/src/assembly/linux.xml
@@ -19,6 +19,10 @@
META-INF/MANIFEST.MF
+
+ libduckdb_java.so_linux_arm64
+ libduckdb_java.so_osx_universal
+ libduckdb_java.so_windows_amd64
/
diff --git a/src/assembly/mac-aarch64.xml b/src/assembly/mac-aarch64.xml
index d9823a4..a89b6de 100644
--- a/src/assembly/mac-aarch64.xml
+++ b/src/assembly/mac-aarch64.xml
@@ -19,6 +19,10 @@
META-INF/MANIFEST.MF
+
+ libduckdb_java.so_linux_amd64
+ libduckdb_java.so_linux_arm64
+ libduckdb_java.so_windows_amd64
/
diff --git a/src/assembly/mac.xml b/src/assembly/mac.xml
index fe206f0..7bf2e2f 100644
--- a/src/assembly/mac.xml
+++ b/src/assembly/mac.xml
@@ -19,6 +19,10 @@
META-INF/MANIFEST.MF
+
+ libduckdb_java.so_linux_amd64
+ libduckdb_java.so_linux_arm64
+ libduckdb_java.so_windows_amd64
/
diff --git a/src/assembly/win.xml b/src/assembly/win.xml
index 9981152..c738f42 100644
--- a/src/assembly/win.xml
+++ b/src/assembly/win.xml
@@ -19,6 +19,10 @@
META-INF/MANIFEST.MF
+
+ libduckdb_java.so_linux_amd64
+ libduckdb_java.so_linux_arm64
+ libduckdb_java.so_osx_universal
/
diff --git a/src/main/java/com/github/sqljam/config/Config.java b/src/main/java/com/github/sqljam/config/Config.java
index 3ea4b78..b832df3 100644
--- a/src/main/java/com/github/sqljam/config/Config.java
+++ b/src/main/java/com/github/sqljam/config/Config.java
@@ -18,6 +18,7 @@
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
+import java.io.InputStreamReader;
import java.io.Reader;
import java.io.Writer;
import java.net.URISyntaxException;
@@ -67,7 +68,7 @@ public Config(File appFile, File userFile) {
this.userFile = userFile;
try (InputStream in = Config.class.getResourceAsStream("/" + FILE_NAME)) {
if (in != null) {
- defaults.load(new java.io.InputStreamReader(in, StandardCharsets.UTF_8));
+ defaults.load(new InputStreamReader(in, StandardCharsets.UTF_8));
}
} catch (IOException e) {
throw new IllegalStateException("Unable to load default configuration", e);
diff --git a/src/main/java/com/github/sqljam/face/SqlJamApplication.java b/src/main/java/com/github/sqljam/face/SqlJamApplication.java
index 17b8ce4..3a78562 100644
--- a/src/main/java/com/github/sqljam/face/SqlJamApplication.java
+++ b/src/main/java/com/github/sqljam/face/SqlJamApplication.java
@@ -15,10 +15,15 @@
*/
package com.github.sqljam.face;
+import java.io.File;
+import java.io.IOException;
+
import com.github.sqljam.config.Config;
import com.github.sqljam.face.model.AppSettings;
import com.github.sqljam.face.model.ConnectionProfile;
import com.github.sqljam.face.service.ConnectionStore;
+import com.github.sqljam.face.service.ExampleDatabase;
+import com.github.sqljam.face.service.JsonStore;
import com.github.sqljam.face.service.SettingsStore;
import com.github.sqljam.face.view.AppContext;
import com.github.sqljam.face.view.Branding;
@@ -68,7 +73,17 @@ public void init() {
settingsStore = new SettingsStore();
notifyPreloader(new SplashPreloader.Status(Messages.get("splash.loadingDataSources"), 0.35));
+ File connectionsFile = Config.getInstance().getFile("sqljam.connections.file",
+ new File(JsonStore.DEFAULT_DIRECTORY, "connections.json"));
profileRegistry = new ProfileRegistry(new ConnectionStore());
+ // A new user starts with an example database of a small online shop
+ if (!connectionsFile.exists()) {
+ notifyPreloader(new SplashPreloader.Status(Messages.get("splash.preparingExample"), 0.38));
+ }
+ ExampleDatabase exampleDatabase = new ExampleDatabase();
+ exampleDatabase.createAtFirstStart(connectionsFile).ifPresent(this::saveExample);
+ // The database file of the example data source may be deleted by the user
+ exampleDatabase.repairAtStart(profileRegistry.getProfiles());
// Loading JDBC drivers takes a while, it is done here instead of at the first connection
DbType[] dbTypes = DbType.values();
@@ -90,6 +105,16 @@ public void init() {
Branding.getIcons();
}
+ private void saveExample(ConnectionProfile profile) {
+ try {
+ profileRegistry.saveProfile(profile, true);
+ } catch (IOException e) {
+ if (log.isWarnEnabled()) {
+ log.warn("Unable to save the data source of the example database", e);
+ }
+ }
+ }
+
@Override
public void start(Stage primaryStage) {
this.stage = primaryStage;
diff --git a/src/main/java/com/github/sqljam/face/model/ConnectionProfile.java b/src/main/java/com/github/sqljam/face/model/ConnectionProfile.java
index 0ca2458..bf19175 100644
--- a/src/main/java/com/github/sqljam/face/model/ConnectionProfile.java
+++ b/src/main/java/com/github/sqljam/face/model/ConnectionProfile.java
@@ -58,8 +58,12 @@ public String getJdbcUrl() {
* Jdbc url connecting to the given database (catalog), PostgreSQL connects to one database per connection
*/
public String getJdbcUrl(String catalog) {
+ if (dbType != null && dbType.isFileBased()) {
+ // The catalog of a database file is the file itself
+ catalog = null;
+ }
if (StringUtils.isNotBlank(url) && (StringUtils.isBlank(catalog) || StringUtils.equals(catalog, database))) {
- return url;
+ return dbType != null ? dbType.normalizeUrl(url) : url;
}
return dbType.getUrl(hostname, port, StringUtils.defaultIfBlank(catalog, database));
}
diff --git a/src/main/java/com/github/sqljam/face/model/TransferRequest.java b/src/main/java/com/github/sqljam/face/model/TransferRequest.java
index ae2317b..1bfc4b0 100644
--- a/src/main/java/com/github/sqljam/face/model/TransferRequest.java
+++ b/src/main/java/com/github/sqljam/face/model/TransferRequest.java
@@ -17,14 +17,19 @@
import java.io.File;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import com.github.sqljam.impexp.DataFileStrategy;
+import com.github.sqljam.impexp.DataFormat;
import com.github.sqljam.impexp.DbType;
import com.github.sqljam.impexp.ExportMode;
import com.github.sqljam.impexp.Exporter;
import com.github.sqljam.impexp.IdentifierCase;
+import com.github.sqljam.impexp.ParquetExporter;
import com.github.sqljam.impexp.ScriptExportHandler;
+import com.github.sqljam.impexp.TableQuery;
import lombok.Getter;
import lombok.Setter;
@@ -81,10 +86,27 @@ public enum Target {
*/
private long maxFileSize = ScriptExportHandler.DEFAULT_MAX_FILE_SIZE;
private boolean lobSeparated = true;
+ /**
+ * Rows of the export package: SQL insert statements or Parquet files
+ */
+ private DataFormat dataFormat = DataFormat.SQL;
+ /**
+ * Compression of Parquet files
+ */
+ private String compression = ParquetExporter.DEFAULT_COMPRESSION;
// Database target
private ConnectionProfile targetProfile;
private String targetCatalog;
private String targetSchema;
private boolean targetSchemaCreated = true;
+ /**
+ * Name pattern of target tables, {table} is the source table name, e.g. {table}_copy copies tables in the same
+ * schema. Target tables have the same names if it is blank.
+ */
+ private String tableNamePattern;
+ /**
+ * Rows and columns of tables narrowed in the data viewer, by table name
+ */
+ private Map tableQueries = new HashMap<>();
}
diff --git a/src/main/java/com/github/sqljam/face/service/DatabaseSession.java b/src/main/java/com/github/sqljam/face/service/DatabaseSession.java
index 470e176..5cd0ebb 100644
--- a/src/main/java/com/github/sqljam/face/service/DatabaseSession.java
+++ b/src/main/java/com/github/sqljam/face/service/DatabaseSession.java
@@ -16,6 +16,7 @@
package com.github.sqljam.face.service;
import java.io.Closeable;
+import java.math.BigDecimal;
import java.sql.Connection;
import java.sql.DatabaseMetaData;
import java.sql.DriverManager;
@@ -48,6 +49,7 @@
import com.github.sqljam.impexp.HikariDataSourceConnectionFactory;
import com.github.sqljam.impexp.MetaDataOperations;
import com.github.sqljam.impexp.TableMetaData;
+import com.github.sqljam.impexp.TableQuery;
import com.github.sqljam.jdbc.ConnectionFactory;
import com.github.sqljam.jdbc.JdbcUtils;
import lombok.extern.slf4j.Slf4j;
@@ -308,24 +310,40 @@ public long countRows(String catalog, String schema, String table) throws SQLExc
*/
public DataPage getData(String catalog, String schema, String table, int pageNumber, int pageSize)
throws SQLException {
- long totalRows = countRows(catalog, schema, table);
+ return getData(catalog, schema, table, null, pageNumber, pageSize);
+ }
+
+ /**
+ * A page of rows narrowed by a query: selected columns, condition, groups and order. Rows are ordered by the
+ * order of the query, then by primary key (group columns of grouped rows).
+ *
+ * @param query null for all rows and columns
+ * @param pageNumber 1-based page number
+ */
+ public DataPage getData(String catalog, String schema, String table, TableQuery query, int pageNumber,
+ int pageSize) throws SQLException {
+ long totalRows;
+ try (Connection connection = getConnection(catalog)) {
+ String countSql = dialect.getCountQueryStatement(getMetaCatalog(catalog), schema, table, query);
+ Long count = JdbcUtils.fetchOne(connection, countSql, Long.class);
+ totalRows = count != null ? count : 0;
+ }
try (Connection connection = getConnection(catalog)) {
Dialect dialect = this.dialect.forVersion(connection.getMetaData().getDatabaseMajorVersion(),
connection.getMetaData().getDatabaseMinorVersion());
List primaryKeys = metaDataOperations.getPrimaryKeyInfos(connection.getMetaData(),
getMetaCatalog(catalog), schema, table).stream()
.sorted(Comparator.comparingInt(info -> TableMetaData.getInt(info, "KEY_SEQ")))
- .map(info -> dialect.quoteIdentifier((String) info.get("COLUMN_NAME")))
- .collect(Collectors.toList());
+ .map(info -> (String) info.get("COLUMN_NAME")).collect(Collectors.toList());
List> columnInfos = new ArrayList<>(metaDataOperations.getColumnInfos(
connection.getMetaData(), getMetaCatalog(catalog), schema, table));
columnInfos.sort(Comparator.comparingInt(info -> TableMetaData.getInt(info, "ORDINAL_POSITION")));
String sql = columnInfos.isEmpty() ? dialect.getSelectTableStatement(getMetaCatalog(catalog), schema,
- table) : dialect.getSelectTableStatement(getMetaCatalog(catalog), schema, table,
+ table) : dialect.getQueryStatement(getMetaCatalog(catalog), schema, table,
columnInfos.stream().map(info -> (String) info.get("COLUMN_NAME")).toArray(String[]::new),
- columnInfos.stream().map(info -> (String) info.get("TYPE_NAME")).toArray(String[]::new));
- String pageSql = dialect.getPageStatement(sql, primaryKeys.isEmpty() ? null
- : String.join(",", primaryKeys), pageSize, Math.max(pageNumber - 1, 0) * pageSize);
+ columnInfos.stream().map(info -> (String) info.get("TYPE_NAME")).toArray(String[]::new), query);
+ String pageSql = dialect.getPageStatement(sql, dialect.getQueryOrderBy(query, primaryKeys), pageSize,
+ Math.max(pageNumber - 1, 0) * pageSize);
List columns = new ArrayList<>();
List> rows = new ArrayList<>();
try (PreparedStatement ps = connection.prepareStatement(pageSql); ResultSet rs = ps.executeQuery()) {
@@ -368,7 +386,7 @@ static String formatValue(Object value) {
if (value instanceof Object[]) {
return Arrays.deepToString((Object[]) value);
}
- String text = value instanceof java.math.BigDecimal ? ((java.math.BigDecimal) value).toPlainString()
+ String text = value instanceof BigDecimal ? ((BigDecimal) value).toPlainString()
: value.toString();
return text.length() > MAX_CELL_LENGTH ? text.substring(0, MAX_CELL_LENGTH) + "..." : text;
}
diff --git a/src/main/java/com/github/sqljam/face/service/ExampleDatabase.java b/src/main/java/com/github/sqljam/face/service/ExampleDatabase.java
new file mode 100644
index 0000000..4b8b1ad
--- /dev/null
+++ b/src/main/java/com/github/sqljam/face/service/ExampleDatabase.java
@@ -0,0 +1,158 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.face.service;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Optional;
+
+import org.apache.commons.io.FileUtils;
+import com.github.sqljam.config.Config;
+import com.github.sqljam.face.model.ConnectionProfile;
+import com.github.sqljam.impexp.DbType;
+import com.github.sqljam.impexp.SqlScriptRunner;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * @Description: ExampleDatabase creates an H2 database of a small online shop at the first start (no data sources are
+ * saved yet), so that a new user can browse, query, export and import tables right away. It is
+ * configured by sqljam.example.enabled and sqljam.example.directory.
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+@Slf4j
+public class ExampleDatabase {
+
+ /**
+ * Name of the data source of the example database
+ */
+ public static final String PROFILE_NAME = "Example Shop (H2)";
+ /**
+ * Id of the data source of the example database, a restored example replaces it
+ */
+ public static final String PROFILE_ID = "example-shop";
+ public static final String DATABASE_NAME = "shop";
+ public static final String USERNAME = "sa";
+
+ private static final String SCRIPT = "/com/github/sqljam/face/example/shop.sql";
+
+ private final File directory;
+ private final boolean enabled;
+
+ public ExampleDatabase() {
+ this(Config.getInstance().getBoolean("sqljam.example.enabled", true), Config.getInstance().getFile(
+ "sqljam.example.directory", new File(JsonStore.DEFAULT_DIRECTORY, "example")));
+ }
+
+ public ExampleDatabase(boolean enabled, File directory) {
+ this.enabled = enabled;
+ this.directory = directory;
+ }
+
+ /**
+ * The example database at the first start: when the file of data sources does not exist yet. A data source
+ * removed by the user is not created again.
+ *
+ * @param connectionsFile file of the saved data sources
+ * @return the data source of the example database, empty if it is not created
+ */
+ public Optional createAtFirstStart(File connectionsFile) {
+ if (!enabled || connectionsFile.exists()) {
+ return Optional.empty();
+ }
+ try {
+ return Optional.of(create());
+ } catch (IOException | SQLException e) {
+ // The example is a convenience, the application starts without it
+ if (log.isWarnEnabled()) {
+ log.warn("Unable to create the example database in {}", directory, e);
+ }
+ return Optional.empty();
+ }
+ }
+
+ /**
+ * The example database of a saved example data source whose database file is missing (deleted by the user),
+ * it is created again at the start
+ *
+ * @return the data source if its database is created again
+ */
+ public Optional repairAtStart(List profiles) {
+ Optional example = profiles.stream().filter(ExampleDatabase::isExample).findFirst();
+ if (!enabled || example.isEmpty() || !isDatabaseMissing(example.get())) {
+ return Optional.empty();
+ }
+ try {
+ return Optional.of(create());
+ } catch (IOException | SQLException e) {
+ if (log.isWarnEnabled()) {
+ log.warn("Unable to create the example database in {} again", directory, e);
+ }
+ return Optional.empty();
+ }
+ }
+
+ public static boolean isExample(ConnectionProfile profile) {
+ return profile != null && PROFILE_ID.equals(profile.getId());
+ }
+
+ /**
+ * Whether the database file of the data source does not exist, H2 would create an empty database
+ */
+ public static boolean isDatabaseMissing(ConnectionProfile profile) {
+ return profile.getDatabase() == null || !new File(profile.getDatabase() + ".mv.db").isFile();
+ }
+
+ /**
+ * Creates the database and its tables, an existing database of the directory is replaced
+ */
+ public ConnectionProfile create() throws IOException, SQLException {
+ FileUtils.forceMkdir(directory);
+ File file = new File(directory, DATABASE_NAME);
+ FileUtils.deleteQuietly(new File(file.getPath() + ".mv.db"));
+ FileUtils.deleteQuietly(new File(file.getPath() + ".trace.db"));
+ ConnectionProfile profile = new ConnectionProfile();
+ profile.setId(PROFILE_ID);
+ profile.setName(PROFILE_NAME);
+ profile.setDbType(DbType.H2);
+ profile.setDatabase(file.getAbsolutePath());
+ profile.setUsername(USERNAME);
+ profile.setPassword("");
+ try (Connection connection = DriverManager.getConnection(profile.getJdbcUrl(), USERNAME, "");
+ InputStream in = ExampleDatabase.class.getResourceAsStream(SCRIPT)) {
+ if (in == null) {
+ throw new IOException("Script of the example database not found: " + SCRIPT);
+ }
+ new SqlScriptRunner(connection).runScript(new InputStreamReader(in, StandardCharsets.UTF_8));
+ }
+ if (log.isInfoEnabled()) {
+ log.info("Example database created: {}", file);
+ }
+ return profile;
+ }
+
+ public File getDirectory() {
+ return directory;
+ }
+}
diff --git a/src/main/java/com/github/sqljam/face/service/TransferService.java b/src/main/java/com/github/sqljam/face/service/TransferService.java
index 7d725fc..cf76c59 100644
--- a/src/main/java/com/github/sqljam/face/service/TransferService.java
+++ b/src/main/java/com/github/sqljam/face/service/TransferService.java
@@ -19,18 +19,25 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.config.Config;
import com.github.sqljam.face.model.ConnectionProfile;
import com.github.sqljam.face.model.TransferRequest;
+import com.github.sqljam.impexp.AbstractFileImporter;
+import com.github.sqljam.impexp.DataFormat;
import com.github.sqljam.impexp.DbType;
+import com.github.sqljam.impexp.DuckDBWorkspace;
import com.github.sqljam.impexp.ExportListener;
import com.github.sqljam.impexp.Exporter;
import com.github.sqljam.impexp.ImportExportHandler;
import com.github.sqljam.impexp.ImportExporter;
+import com.github.sqljam.impexp.ParquetExporter;
+import com.github.sqljam.impexp.ParquetImporter;
import com.github.sqljam.impexp.ScriptExporter;
-import com.github.sqljam.impexp.ScriptImporter;
import com.github.sqljam.impexp.SqlScriptRunner;
/**
@@ -66,12 +73,17 @@ private void configureSource(Exporter.ExportConfiguration configuration, Transfe
configuration.setPageSize(request.getPageSize());
configuration.setLobPageSize(request.getLobPageSize());
configuration.setConnectionPoolEnabled(true);
+ configuration.setTableQueries(new HashMap<>(request.getTableQueries()));
}
private void exportScripts(TransferRequest request, ExportListener listener) throws Exception {
if (request.getOutputDirectory() == null) {
throw new IllegalArgumentException("Output directory must be required.");
}
+ if (request.getDataFormat() == DataFormat.PARQUET) {
+ exportParquet(request, listener);
+ return;
+ }
ScriptExporter scriptExporter = new ScriptExporter(request.getOutputDirectory(),
request.getDataFileStrategy(), request.getMaxFileSize());
configureSource(scriptExporter.getConfiguration(), request);
@@ -87,6 +99,104 @@ private void exportScripts(TransferRequest request, ExportListener listener) thr
scriptExporter.export(request.getExportMode());
}
+ /**
+ * Export package with Parquet files: schema.sql and constraints.sql for the target database type
+ */
+ private void exportParquet(TransferRequest request, ExportListener listener) throws Exception {
+ ParquetExporter parquetExporter = new ParquetExporter(request.getOutputDirectory());
+ configureSource(parquetExporter.getConfiguration(), request);
+ parquetExporter.setTargetDbType(request.getScriptDbType());
+ parquetExporter.setTargetSchemaName(request.getScriptSchema());
+ parquetExporter.setIdentifierCase(request.getIdentifierCase());
+ parquetExporter.setCompression(request.getCompression());
+ int[] version = parseVersion(request.getScriptDbVersion());
+ if (version != null) {
+ parquetExporter.setTargetVersion(version[0], version[1]);
+ }
+ parquetExporter.setExportListener(listener);
+ parquetExporter.export(request.getExportMode());
+ }
+
+ /**
+ * Imports Parquet files of other tools into a table: a new table created from the Parquet schema, or rows
+ * appended to an existing table
+ */
+ public ParquetImporter importParquet(ConnectionProfile target, String catalog, String schema, List files,
+ String tableName, ParquetImporter.Mode mode, ExportListener listener)
+ throws Exception {
+ String url = target.getJdbcUrl(StringUtils.defaultIfBlank(catalog, null));
+ try (Connection connection = StringUtils.isBlank(target.getUsername()) ? DriverManager.getConnection(url)
+ : DriverManager.getConnection(url, target.getUsername(), target.getPassword())) {
+ if (StringUtils.isNotBlank(schema) && target.getDbType().isCanSetSchema()) {
+ connection.setSchema(schema);
+ }
+ if (StringUtils.isNotBlank(catalog) && (target.getDbType() == DbType.MYSQL
+ || target.getDbType() == DbType.MARIADB)) {
+ connection.setCatalog(catalog);
+ }
+ ParquetImporter importer = new ParquetImporter(connection, target.getDbType());
+ if (StringUtils.isNotBlank(schema) && target.getDbType().isSchemaSupported()) {
+ importer.setTargetSchemaName(schema);
+ }
+ importer.setPageSize(Config.getInstance().getInt("sqljam.export.page-size", Exporter.DEFAULT_PAGE_SIZE));
+ importer.setExportListener(listener);
+ importer.importFiles(files, tableName, mode);
+ return importer;
+ }
+ }
+
+ /**
+ * Columns (name and type) and rows of Parquet files, shown before importing
+ */
+ public ParquetPreview previewParquet(List files) throws Exception {
+ return previewParquet(files, 0);
+ }
+
+ /**
+ * Columns, rows and the first rows (values as text) of Parquet files
+ */
+ public ParquetPreview previewParquet(List files, int rowLimit) throws Exception {
+ try (DuckDBWorkspace workspace = new DuckDBWorkspace(null)) {
+ List> firstRows = rowLimit > 0 ? workspace.readParquetRows(files, rowLimit)
+ : Collections.emptyList();
+ return new ParquetPreview(workspace.describeParquet(files), workspace.countParquetRows(files), firstRows);
+ }
+ }
+
+ /**
+ * Columns and rows of Parquet files
+ */
+ public static class ParquetPreview {
+
+ private final List columns;
+ private final long rows;
+ private final List> firstRows;
+
+ public ParquetPreview(List columns, long rows, List> firstRows) {
+ this.columns = columns;
+ this.rows = rows;
+ this.firstRows = firstRows;
+ }
+
+ /**
+ * First rows of the files, values as text
+ */
+ public List> getFirstRows() {
+ return firstRows;
+ }
+
+ /**
+ * Name and DuckDB type of each column
+ */
+ public List getColumns() {
+ return columns;
+ }
+
+ public long getRows() {
+ return rows;
+ }
+ }
+
private void importDatabase(TransferRequest request, ExportListener listener) throws Exception {
ConnectionProfile target = request.getTargetProfile();
if (target == null) {
@@ -102,6 +212,7 @@ private void importDatabase(TransferRequest request, ExportListener listener) th
importConfiguration.setTargetCatalogName(request.getTargetCatalog());
importConfiguration.setTargetSchemaName(request.getTargetSchema());
importConfiguration.setTargetSchemaCreated(request.isTargetSchemaCreated());
+ importConfiguration.setTableNamePattern(request.getTableNamePattern());
importConfiguration.setConnectionPoolEnabled(true);
importExporter.setIdentifierCase(request.getIdentifierCase());
importExporter.setExportListener(listener);
@@ -109,9 +220,9 @@ private void importDatabase(TransferRequest request, ExportListener listener) th
}
/**
- * Imports a directory exported as sql scripts into the target database
+ * Imports an export package (sql scripts or Parquet files) into the target database
*/
- public ScriptImporter importScripts(ConnectionProfile target, String catalog, String schema, File directory,
+ public AbstractFileImporter importScripts(ConnectionProfile target, String catalog, String schema, File directory,
boolean stopOnError, ExportListener listener) throws Exception {
String url = target.getJdbcUrl(StringUtils.defaultIfBlank(catalog, null));
try (Connection connection = StringUtils.isBlank(target.getUsername()) ? DriverManager.getConnection(url)
@@ -119,10 +230,12 @@ public ScriptImporter importScripts(ConnectionProfile target, String catalog, St
if (StringUtils.isNotBlank(schema) && target.getDbType().isCanSetSchema()) {
connection.setSchema(schema);
}
- if (StringUtils.isNotBlank(catalog) && target.getDbType() == DbType.MYSQL) {
+ if (StringUtils.isNotBlank(catalog) && (target.getDbType() == DbType.MYSQL
+ || target.getDbType() == DbType.MARIADB)) {
connection.setCatalog(catalog);
}
- ScriptImporter importer = new ScriptImporter(connection, target.getDbType());
+ AbstractFileImporter importer = AbstractFileImporter.forDirectory(directory, connection,
+ target.getDbType());
importer.setStopOnError(stopOnError);
importer.setBatchSize(Config.getInstance().getInt("sqljam.import.batch-size",
SqlScriptRunner.DEFAULT_BATCH_SIZE));
diff --git a/src/main/java/com/github/sqljam/face/view/AppContext.java b/src/main/java/com/github/sqljam/face/view/AppContext.java
index 690f004..dbd645a 100644
--- a/src/main/java/com/github/sqljam/face/view/AppContext.java
+++ b/src/main/java/com/github/sqljam/face/view/AppContext.java
@@ -16,12 +16,16 @@
package com.github.sqljam.face.view;
import java.io.IOException;
+import java.sql.SQLException;
import com.github.sqljam.face.model.AppSettings;
+import com.github.sqljam.face.model.ConnectionProfile;
+import com.github.sqljam.face.service.ExampleDatabase;
import com.github.sqljam.face.service.SettingsStore;
import com.github.sqljam.face.service.TransferService;
import javafx.application.HostServices;
import lombok.Getter;
+import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
/**
@@ -39,6 +43,8 @@ public class AppContext {
private final SessionManager sessionManager = new SessionManager();
private final TransferService transferService = new TransferService();
private final HostServices hostServices;
+ @Setter
+ private ExampleDatabase exampleDatabase = new ExampleDatabase();
public AppContext(SettingsStore settingsStore, ProfileRegistry profileRegistry, HostServices hostServices) {
this.settingsStore = settingsStore;
@@ -46,6 +52,26 @@ public AppContext(SettingsStore settingsStore, ProfileRegistry profileRegistry,
this.hostServices = hostServices;
}
+ /**
+ * Creates the example database again and saves its data source, e.g. after the user removed it. Sessions of
+ * the example are closed before, so that its file is not locked. It runs in the background.
+ */
+ public ConnectionProfile restoreExample() throws IOException, SQLException {
+ sessionManager.closeSessionNow(ExampleDatabase.PROFILE_ID);
+ ConnectionProfile profile = exampleDatabase.create();
+ profileRegistry.saveProfile(profile, true);
+ return profile;
+ }
+
+ /**
+ * Opens the url in the browser of the system, nothing happens without host services, e.g. in tests
+ */
+ public void openDocument(String url) {
+ if (hostServices != null) {
+ hostServices.showDocument(url);
+ }
+ }
+
public AppSettings getSettings() {
return settingsStore.getSettings();
}
diff --git a/src/main/java/com/github/sqljam/face/view/Branding.java b/src/main/java/com/github/sqljam/face/view/Branding.java
index 7b414c1..f3e5ea1 100644
--- a/src/main/java/com/github/sqljam/face/view/Branding.java
+++ b/src/main/java/com/github/sqljam/face/view/Branding.java
@@ -21,8 +21,12 @@
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Consumer;
+import javafx.scene.Cursor;
import javafx.scene.control.Dialog;
+import javafx.scene.control.Label;
+import javafx.scene.control.Tooltip;
import javafx.scene.image.Image;
import javafx.scene.image.ImageView;
import javafx.stage.Stage;
@@ -44,6 +48,16 @@ public final class Branding {
public static final String LOGO = "logo.png";
public static final String LOGO_MARK = "logo-mark.png";
public static final int[] ICON_SIZES = {16, 24, 32, 48, 64, 128, 256, 512};
+ /**
+ * Source repository of SqlJam
+ */
+ public static final String REPOSITORY_URL = "https://github.com/paganini2008/sqljam";
+ /**
+ * Home page of SqlJam, the GitHub Pages of the repository
+ */
+ public static final String HOMEPAGE_URL = "https://paganini2008.github.io/sqljam/";
+ public static final String AUTHOR = "Fred Feng";
+ public static final String EMAIL = "paganini20080@gmail.com";
private static final Map images = new ConcurrentHashMap<>();
private static volatile List icons;
@@ -51,6 +65,23 @@ public final class Branding {
private Branding() {
}
+ /**
+ * GitHub mark of the source repository, the url is shown by its tooltip
+ *
+ * @param browser opens the repository in the browser when the mark is clicked, null for a mark without a click
+ */
+ public static Label repositoryMark(Consumer browser) {
+ Label mark = new Label(null, Icons.of(Icons.GITHUB));
+ mark.setId("repositoryMark");
+ mark.getStyleClass().add("repository-mark");
+ mark.setTooltip(new Tooltip(REPOSITORY_URL));
+ if (browser != null) {
+ mark.setCursor(Cursor.HAND);
+ mark.setOnMouseClicked(event -> browser.accept(REPOSITORY_URL));
+ }
+ return mark;
+ }
+
/**
* Returns the image in the image directory, or null if it does not exist
*/
diff --git a/src/main/java/com/github/sqljam/face/view/ConnectionDialog.java b/src/main/java/com/github/sqljam/face/view/ConnectionDialog.java
index 85de6ad..3af535c 100644
--- a/src/main/java/com/github/sqljam/face/view/ConnectionDialog.java
+++ b/src/main/java/com/github/sqljam/face/view/ConnectionDialog.java
@@ -17,13 +17,19 @@
import java.io.IOException;
import java.util.Optional;
+import java.util.function.Consumer;
import com.github.sqljam.face.model.ConnectionProfile;
import javafx.event.ActionEvent;
+import javafx.geometry.Pos;
import javafx.scene.control.Button;
import javafx.scene.control.ButtonBar;
import javafx.scene.control.ButtonType;
+import javafx.scene.control.ComboBox;
import javafx.scene.control.Dialog;
+import javafx.scene.control.Tooltip;
+import javafx.scene.layout.HBox;
+import javafx.scene.layout.Priority;
import javafx.stage.Window;
/**
@@ -74,6 +80,32 @@ public ConnectionDialog(Window owner, AppContext context, ConnectionProfile prof
setResultConverter(buttonType -> buttonType == ButtonType.OK ? form.getProfile() : null);
}
+ /**
+ * Target data source of an import: the combo box and a button creating a new data source, which is passed to
+ * the consumer to be listed and selected
+ */
+ public static HBox targetField(ComboBox combo, AppContext context,
+ Consumer onCreated) {
+ Button newButton = new Button(null, Icons.of(Icons.ADD));
+ newButton.setId("newTargetButton");
+ newButton.setTooltip(new Tooltip(Messages.get("action.newTargetConnection")));
+ newButton.setOnAction(event -> show(combo.getScene() != null ? combo.getScene().getWindow() : null, context,
+ null).ifPresent(onCreated));
+ combo.setMaxWidth(Double.MAX_VALUE);
+ HBox.setHgrow(combo, Priority.ALWAYS);
+ HBox field = new HBox(6, combo, newButton);
+ field.setAlignment(Pos.CENTER_LEFT);
+ return field;
+ }
+
+ /**
+ * Selects the data source of the same id in the combo box, the selection is kept if it is not listed
+ */
+ public static void selectProfile(ComboBox combo, ConnectionProfile profile) {
+ combo.getItems().stream().filter(item -> item.getId().equals(profile.getId())).findFirst()
+ .ifPresent(combo::setValue);
+ }
+
public static Optional show(Window owner, AppContext context, ConnectionProfile profile) {
return new ConnectionDialog(owner, context, profile).showAndWait();
}
diff --git a/src/main/java/com/github/sqljam/face/view/ConnectionForm.java b/src/main/java/com/github/sqljam/face/view/ConnectionForm.java
index 5ec0501..e7b604c 100644
--- a/src/main/java/com/github/sqljam/face/view/ConnectionForm.java
+++ b/src/main/java/com/github/sqljam/face/view/ConnectionForm.java
@@ -17,6 +17,7 @@
import java.io.File;
import java.util.Objects;
+import java.util.function.Consumer;
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.face.model.ConnectionProfile;
@@ -25,18 +26,22 @@
import javafx.beans.property.BooleanProperty;
import javafx.beans.property.SimpleBooleanProperty;
import javafx.geometry.Insets;
+import javafx.geometry.Pos;
import javafx.scene.control.Button;
import javafx.scene.control.CheckBox;
+import javafx.scene.control.ChoiceDialog;
import javafx.scene.control.ComboBox;
import javafx.scene.control.Label;
import javafx.scene.control.PasswordField;
import javafx.scene.control.ProgressIndicator;
import javafx.scene.control.TextField;
import javafx.scene.control.TextFormatter;
+import javafx.scene.control.Tooltip;
import javafx.scene.layout.ColumnConstraints;
import javafx.scene.layout.GridPane;
import javafx.scene.layout.HBox;
import javafx.scene.layout.Priority;
+import javafx.scene.layout.Region;
import javafx.scene.layout.VBox;
import javafx.stage.FileChooser;
@@ -85,7 +90,8 @@ public ConnectionForm() {
urlField.setId("urlField");
statusLabel.setId("formStatusLabel");
- dbTypeCombo.getItems().addAll(DbType.values());
+ dbTypeCombo.getItems().addAll(DbTypeCells.getDbTypesByCategory());
+ DbTypeCells.setupDbTypeCombo(dbTypeCombo);
dbTypeCombo.setMaxWidth(Double.MAX_VALUE);
dbTypeCombo.valueProperty().addListener((obs, oldType, newType) -> onDbTypeChanged(oldType, newType));
portField.setTextFormatter(new TextFormatter(change -> change.getControlNewText().matches("\\d{0,5}")
@@ -109,8 +115,8 @@ public ConnectionForm() {
progressIndicator.managedProperty().bind(busy);
hostBox = new HBox(8, hostField, portLabel, portField);
- hostBox.setAlignment(javafx.geometry.Pos.CENTER_LEFT);
- portLabel.setMinWidth(javafx.scene.layout.Region.USE_PREF_SIZE);
+ hostBox.setAlignment(Pos.CENTER_LEFT);
+ portLabel.setMinWidth(Region.USE_PREF_SIZE);
HBox.setHgrow(hostField, Priority.ALWAYS);
HBox databaseBox = new HBox(8, databaseField, browseButton);
HBox.setHgrow(databaseField, Priority.ALWAYS);
@@ -148,8 +154,8 @@ private void onDbTypeChanged(DbType oldType, DbType newType) {
portField.setText(newType.getDefaultPort() > 0 ? String.valueOf(newType.getDefaultPort()) : "");
}
boolean fileBased = newType.isFileBased();
- // H2 may run as a tcp server, SQLite is always a file
- boolean hostVisible = newType != DbType.SQLITE;
+ // H2 may run as a tcp server, SQLite and DuckDB are always files
+ boolean hostVisible = !fileBased || newType == DbType.H2;
hostLabel.setVisible(hostVisible);
hostLabel.setManaged(hostVisible);
hostBox.setVisible(hostVisible);
@@ -160,7 +166,7 @@ private void onDbTypeChanged(DbType oldType, DbType newType) {
browseButton.setVisible(browsable);
browseButton.setManaged(browsable);
browseButton.setGraphic(Icons.of(fileBased ? Icons.FOLDER : "fth-more-horizontal"));
- browseButton.setTooltip(fileBased ? null : new javafx.scene.control.Tooltip(
+ browseButton.setTooltip(fileBased ? null : new Tooltip(
Messages.get("connection.browseDatabases")));
if (newType == DbType.ORACLE) {
databaseLabel.setText(Messages.get("connection.serviceName"));
@@ -213,7 +219,7 @@ private void browseDatabases() {
return;
}
String current = databases.contains(databaseField.getText()) ? databaseField.getText() : databases.get(0);
- javafx.scene.control.ChoiceDialog dialog = new javafx.scene.control.ChoiceDialog<>(current,
+ ChoiceDialog dialog = new ChoiceDialog<>(current,
databases);
dialog.initOwner(getScene() != null ? getScene().getWindow() : null);
Branding.applyIcons(dialog);
@@ -307,8 +313,8 @@ public String validate() {
if (StringUtils.isNotBlank(edited.getUrl())) {
return null;
}
- boolean fileMode = dbType == DbType.SQLITE || (dbType == DbType.H2 && StringUtils.isBlank(
- edited.getHostname()));
+ boolean fileMode = (dbType.isFileBased() && dbType != DbType.H2) || (dbType == DbType.H2
+ && StringUtils.isBlank(edited.getHostname()));
if (!fileMode && StringUtils.isBlank(edited.getHostname())) {
return Messages.get("connection.error.host");
}
@@ -352,7 +358,7 @@ public void clearStatus() {
*
* @param onSuccess called with product name and version
*/
- public void testConnection(java.util.function.Consumer onSuccess) {
+ public void testConnection(Consumer onSuccess) {
String error = validate();
if (error != null) {
showError(error);
diff --git a/src/main/java/com/github/sqljam/face/view/ConnectionTree.java b/src/main/java/com/github/sqljam/face/view/ConnectionTree.java
index da8955f..c5c95c9 100644
--- a/src/main/java/com/github/sqljam/face/view/ConnectionTree.java
+++ b/src/main/java/com/github/sqljam/face/view/ConnectionTree.java
@@ -24,9 +24,11 @@
import com.github.sqljam.face.model.TableInfo;
import com.github.sqljam.face.service.DatabaseSession;
import com.github.sqljam.impexp.DbType;
+import javafx.application.Platform;
import javafx.scene.control.ContextMenu;
import javafx.scene.control.MenuItem;
import javafx.scene.control.SeparatorMenuItem;
+import javafx.scene.control.Tooltip;
import javafx.scene.control.TreeCell;
import javafx.scene.control.TreeItem;
import javafx.scene.control.TreeView;
@@ -52,11 +54,15 @@ public class ConnectionTree extends TreeView {
@Setter
private Consumer onImportScripts;
@Setter
+ private Consumer onImportParquet;
+ @Setter
private Consumer onEditConnection;
@Setter
private Consumer onDeleteConnection;
@Setter
private Consumer onStatus;
+ @Setter
+ private Consumer onDisconnect;
public ConnectionTree(AppContext context) {
this.context = context;
@@ -78,6 +84,46 @@ public void reloadConnections() {
}
}
+ /**
+ * Adds connections of saved profiles which are not in the tree yet, e.g. created as the target of an import,
+ * opened connections are kept
+ */
+ public void addNewConnections() {
+ for (ConnectionProfile profile : context.getProfileRegistry().getProfiles()) {
+ boolean exists = rootItem.getChildren().stream().anyMatch(item -> item.getValue().getProfile().getId()
+ .equals(profile.getId()));
+ if (!exists) {
+ rootItem.getChildren().add(new LazyItem(DbNode.connection(profile)));
+ }
+ }
+ }
+
+ /**
+ * Whether the connection of the item is opened: its children are loaded
+ */
+ public boolean isConnected(TreeItem item) {
+ return item instanceof LazyItem && item.getValue().getKind() == DbNode.Kind.CONNECTION
+ && ((LazyItem) item).loaded;
+ }
+
+ /**
+ * Closes the session of the connection, the connection is opened again when it is expanded
+ */
+ public void disconnect(TreeItem item) {
+ if (!isConnected(item)) {
+ return;
+ }
+ ConnectionProfile profile = item.getValue().getProfile();
+ ((LazyItem) item).unload();
+ context.getSessionManager().closeSession(profile.getId());
+ if (onDisconnect != null) {
+ onDisconnect.accept(profile);
+ }
+ if (onStatus != null) {
+ onStatus.accept(Messages.format("tree.disconnected", profile.getName()));
+ }
+ }
+
public DbNode getSelectedNode() {
TreeItem item = getSelectionModel().getSelectedItem();
return item != null ? item.getValue() : null;
@@ -120,7 +166,7 @@ private List> loadChildren(DbNode node) throws Exception {
case CONNECTION:
String product = session.getDatabaseProduct();
if (onStatus != null) {
- javafx.application.Platform.runLater(() -> onStatus.accept(profile.getName() + " - " + product));
+ Platform.runLater(() -> onStatus.accept(profile.getName() + " - " + product));
}
// Databases of the server, Oracle and SQLite have no databases
List catalogs = session.getCatalogs();
@@ -182,6 +228,15 @@ public boolean isLeaf() {
return false;
}
+ /**
+ * Collapsed with a placeholder child, the children are loaded again when it is expanded
+ */
+ void unload() {
+ loaded = false;
+ setExpanded(false);
+ getChildren().setAll(Collections.singletonList(new TreeItem<>(DbNode.loading())));
+ }
+
void reload() {
loaded = false;
getChildren().setAll(Collections.singletonList(new TreeItem<>(DbNode.loading())));
@@ -255,7 +310,7 @@ protected void updateItem(DbNode node, boolean empty) {
break;
}
if (node.getKind() == DbNode.Kind.TABLE && node.getTable().getRemarks() != null) {
- setTooltip(new javafx.scene.control.Tooltip(node.getTable().getRemarks()));
+ setTooltip(new Tooltip(node.getTable().getRemarks()));
} else {
setTooltip(null);
}
@@ -266,12 +321,23 @@ private ContextMenu createContextMenu(DbNode node) {
ContextMenu menu = new ContextMenu();
switch (node.getKind()) {
case CONNECTION:
- menu.getItems().addAll(
- menuItem("tree.connect", Icons.CONNECT, () -> getTreeItem().setExpanded(true)),
+ // Connect before the connection is opened, disconnect after it is opened
+ MenuItem connect = menuItem("tree.connect", Icons.CONNECT, () -> getTreeItem().setExpanded(true));
+ connect.setId("connectMenuItem");
+ MenuItem disconnect = menuItem("tree.disconnect", Icons.DISCONNECT, () -> disconnect(
+ getTreeItem()));
+ disconnect.setId("disconnectMenuItem");
+ menu.setOnShowing(event -> {
+ boolean connected = isConnected(getTreeItem());
+ connect.setVisible(!connected);
+ disconnect.setVisible(connected);
+ });
+ menu.getItems().addAll(connect, disconnect,
menuItem("action.refresh", Icons.REFRESH, ConnectionTree.this::refreshSelected),
new SeparatorMenuItem(),
menuItem("action.export", Icons.EXPORT, () -> fire(onExport, node)),
menuItem("action.importScripts", Icons.IMPORT, () -> fire(onImportScripts, node)),
+ menuItem("action.importParquet", Icons.IMPORT, () -> fire(onImportParquet, node)),
new SeparatorMenuItem(),
menuItem("action.editConnection", Icons.EDIT, () -> fire(onEditConnection, node)),
menuItem("action.deleteConnection", Icons.DELETE, () -> fire(onDeleteConnection, node)));
@@ -282,7 +348,8 @@ private ContextMenu createContextMenu(DbNode node) {
menuItem("action.refresh", Icons.REFRESH, ConnectionTree.this::refreshSelected),
new SeparatorMenuItem(),
menuItem("action.export", Icons.EXPORT, () -> fire(onExport, node)),
- menuItem("action.importScripts", Icons.IMPORT, () -> fire(onImportScripts, node)));
+ menuItem("action.importScripts", Icons.IMPORT, () -> fire(onImportScripts, node)),
+ menuItem("action.importParquet", Icons.IMPORT, () -> fire(onImportParquet, node)));
break;
case TABLE:
menu.getItems().addAll(
diff --git a/src/main/java/com/github/sqljam/face/view/DataPane.java b/src/main/java/com/github/sqljam/face/view/DataPane.java
index c51bf22..354c2b3 100644
--- a/src/main/java/com/github/sqljam/face/view/DataPane.java
+++ b/src/main/java/com/github/sqljam/face/view/DataPane.java
@@ -15,26 +15,44 @@
*/
package com.github.sqljam.face.view;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Map;
+import java.util.function.Consumer;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import com.github.sqljam.face.model.ColumnInfo;
import com.github.sqljam.face.model.DataPage;
import com.github.sqljam.face.service.DatabaseSession;
+import com.github.sqljam.impexp.TableQuery;
import javafx.beans.property.ReadOnlyObjectWrapper;
import javafx.geometry.Insets;
import javafx.geometry.Pos;
import javafx.scene.control.Button;
+import javafx.scene.control.CheckBox;
import javafx.scene.control.ComboBox;
+import javafx.scene.control.CustomMenuItem;
import javafx.scene.control.Label;
+import javafx.scene.control.MenuButton;
+import javafx.scene.control.SeparatorMenuItem;
import javafx.scene.control.TableCell;
import javafx.scene.control.TableColumn;
import javafx.scene.control.TableView;
+import javafx.scene.control.TextField;
+import javafx.scene.control.Tooltip;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.HBox;
import javafx.scene.layout.Priority;
import javafx.scene.layout.Region;
+import javafx.scene.layout.VBox;
/**
- * @Description: DataPane shows rows of a table page by page
+ * @Description: DataPane shows rows of a table page by page. Rows are narrowed by a simple query: selected
+ * columns, a WHERE condition, GROUP BY columns (with the count of rows) and ORDER BY. The narrowed
+ * rows can be exported (grouped rows are for viewing).
* @Author: Fred Feng
* @Date: 26/03/2023
* @Version 1.0.0
@@ -56,6 +74,19 @@ public class DataPane extends BorderPane {
private final Label pageLabel = new Label();
private final Label totalLabel = new Label();
private final ComboBox pageSizeCombo = new ComboBox<>();
+ private final MenuButton columnsButton = new MenuButton();
+ private final MenuButton groupByButton = new MenuButton();
+ private final TextField whereField = new TextField();
+ private final TextField orderField = new TextField();
+ private final Button applyButton = new Button(Messages.get("data.query.apply"), Icons.of(Icons.START));
+ private final Button clearButton = new Button(Messages.get("data.query.clear"), Icons.of(Icons.CANCEL));
+ private final Button exportButton = new Button(Messages.get("action.export"), Icons.of(Icons.EXPORT));
+ private final Label queryErrorLabel = new Label();
+ private final Map columnChecks = new LinkedHashMap<>();
+ private final Map groupChecks = new LinkedHashMap<>();
+ private TableQuery query = new TableQuery();
+ private Consumer onExport;
+ private boolean columnsLoaded;
private int pageNumber = 1;
private int totalPages = 1;
private List columns;
@@ -92,21 +123,189 @@ public DataPane(DatabaseSession session, String catalog, String schema, String t
pager.setAlignment(Pos.CENTER_LEFT);
pager.setPadding(new Insets(8));
pager.getStyleClass().add("pager");
+ setTop(createQueryBar());
setCenter(dataTable);
setBottom(pager);
updatePager(false);
}
+ /**
+ * Columns, WHERE, GROUP BY and ORDER BY of the rows, applied by Apply or Enter
+ */
+ private VBox createQueryBar() {
+ columnsButton.setId("columnsButton");
+ groupByButton.setId("groupByButton");
+ whereField.setId("whereField");
+ orderField.setId("orderField");
+ applyButton.setId("applyQueryButton");
+ clearButton.setId("clearQueryButton");
+ exportButton.setId("exportQueryButton");
+ queryErrorLabel.setId("queryErrorLabel");
+ whereField.setPromptText(Messages.get("data.query.where"));
+ orderField.setPromptText(Messages.get("data.query.orderBy"));
+ whereField.setOnAction(event -> applyQuery());
+ orderField.setOnAction(event -> applyQuery());
+ applyButton.setOnAction(event -> applyQuery());
+ clearButton.setOnAction(event -> clearQuery());
+ exportButton.setOnAction(event -> {
+ if (onExport != null) {
+ onExport.accept(query);
+ }
+ });
+ HBox.setHgrow(whereField, Priority.ALWAYS);
+ orderField.setPrefWidth(200);
+ updateQueryButtons();
+ HBox bar = new HBox(6, columnsButton, whereField, groupByButton, orderField, applyButton, clearButton,
+ exportButton);
+ bar.setAlignment(Pos.CENTER_LEFT);
+ queryErrorLabel.getStyleClass().add("status-error");
+ queryErrorLabel.setWrapText(true);
+ queryErrorLabel.setVisible(false);
+ queryErrorLabel.setManaged(false);
+ VBox top = new VBox(4, bar, queryErrorLabel);
+ top.setPadding(new Insets(8, 8, 0, 8));
+ top.getStyleClass().add("query-bar");
+ return top;
+ }
+
+ /**
+ * Rows narrowed by the query are exported by the consumer, e.g. the export dialog
+ */
+ public void setOnExport(Consumer onExport) {
+ this.onExport = onExport;
+ }
+
+ public TableQuery getQuery() {
+ return query;
+ }
+
+ /**
+ * Columns of the table for the column and group choices
+ */
+ void setColumns(List columnNames) {
+ columnChecks.clear();
+ groupChecks.clear();
+ columnsButton.getItems().clear();
+ groupByButton.getItems().clear();
+ CheckBox allCheck = new CheckBox(Messages.get("data.query.allColumns"));
+ allCheck.setId("allColumnsCheck");
+ allCheck.setSelected(true);
+ allCheck.setOnAction(event -> {
+ // Checks of columns update the All check, its state is read first
+ boolean all = allCheck.isSelected();
+ columnChecks.values().forEach(check -> check.setSelected(all));
+ });
+ columnsButton.getItems().addAll(menuItem(allCheck), new SeparatorMenuItem());
+ for (String columnName : columnNames) {
+ CheckBox columnCheck = new CheckBox(columnName);
+ columnCheck.setSelected(true);
+ columnCheck.selectedProperty().addListener((obs, oldValue, value) -> {
+ allCheck.setSelected(columnChecks.values().stream().allMatch(CheckBox::isSelected));
+ updateQueryButtons();
+ });
+ columnChecks.put(columnName, columnCheck);
+ columnsButton.getItems().add(menuItem(columnCheck));
+ CheckBox groupCheck = new CheckBox(columnName);
+ groupCheck.selectedProperty().addListener((obs, oldValue, value) -> updateQueryButtons());
+ groupChecks.put(columnName, groupCheck);
+ groupByButton.getItems().add(menuItem(groupCheck));
+ }
+ columnsLoaded = true;
+ updateQueryButtons();
+ }
+
+ private static CustomMenuItem menuItem(CheckBox check) {
+ CustomMenuItem item = new CustomMenuItem(check);
+ // The menu stays open while columns are checked
+ item.setHideOnClick(false);
+ return item;
+ }
+
+ /**
+ * Query of the inputs, all checked columns mean all columns
+ */
+ TableQuery buildQuery() {
+ List columns = columnChecks.values().stream().allMatch(CheckBox::isSelected) ? new ArrayList<>()
+ : columnChecks.entrySet().stream().filter(entry -> entry.getValue().isSelected())
+ .map(Map.Entry::getKey).collect(Collectors.toList());
+ List groupBy = groupChecks.entrySet().stream().filter(entry -> entry.getValue().isSelected())
+ .map(Map.Entry::getKey).collect(Collectors.toList());
+ return new TableQuery(columns, whereField.getText(), groupBy, orderField.getText());
+ }
+
+ void applyQuery() {
+ TableQuery newQuery;
+ try {
+ newQuery = buildQuery();
+ } catch (IllegalArgumentException e) {
+ showQueryError(e.getMessage());
+ return;
+ }
+ if (!newQuery.isGrouped() && !columnChecks.isEmpty() && columnChecks.values().stream()
+ .noneMatch(CheckBox::isSelected)) {
+ showQueryError(Messages.get("data.query.noColumns"));
+ return;
+ }
+ showQueryError(null);
+ query = newQuery;
+ columns = null;
+ updateQueryButtons();
+ load(1);
+ }
+
+ void clearQuery() {
+ whereField.clear();
+ orderField.clear();
+ columnChecks.values().forEach(check -> check.setSelected(true));
+ groupChecks.values().forEach(check -> check.setSelected(false));
+ applyQuery();
+ }
+
+ private void showQueryError(String message) {
+ queryErrorLabel.setText(message != null ? message : "");
+ queryErrorLabel.setVisible(message != null);
+ queryErrorLabel.setManaged(message != null);
+ }
+
+ private void updateQueryButtons() {
+ long selectedColumns = columnChecks.values().stream().filter(CheckBox::isSelected).count();
+ columnsButton.setText(selectedColumns == columnChecks.size() ? Messages.get("data.query.allColumns")
+ : Messages.format("data.query.columns", selectedColumns, columnChecks.size()));
+ List groups = groupChecks.entrySet().stream().filter(entry -> entry.getValue().isSelected())
+ .map(Map.Entry::getKey).collect(Collectors.toList());
+ groupByButton.setText(groups.isEmpty() ? Messages.get("data.query.groupBy")
+ : Messages.format("data.query.groupedBy", String.join(", ", groups)));
+ // Grouped rows have the group columns and the count
+ columnsButton.setDisable(!groups.isEmpty() || !columnsLoaded);
+ groupByButton.setDisable(!columnsLoaded);
+ exportButton.setDisable(query.isGrouped());
+ exportButton.setTooltip(new Tooltip(Messages.get(query.isGrouped() ? "data.query.exportGrouped"
+ : "data.query.export")));
+ }
+
public void load(int page) {
int targetPage = Math.max(1, Math.min(page, Math.max(totalPages, 1)));
int pageSize = pageSizeCombo.getValue();
+ TableQuery currentQuery = query;
updatePager(true);
- TaskRunner.run(() -> session.getData(catalog, schema, table, targetPage, pageSize), this::show, e -> {
- updatePager(false);
- dataTable.setPlaceholder(new Label(Messages.format("tree.error", e.getMessage())));
- Dialogs.showError(getScene() != null ? getScene().getWindow() : null,
- Messages.format("table.loadError", table), e);
- });
+ if (!columnsLoaded) {
+ TaskRunner.run(() -> session.getColumns(catalog, schema, table), columnInfos -> setColumns(
+ columnInfos.stream().map(ColumnInfo::getName).collect(Collectors.toList())), e -> {
+ });
+ }
+ TaskRunner.run(() -> session.getData(catalog, schema, table, currentQuery, targetPage, pageSize),
+ this::show, e -> {
+ updatePager(false);
+ dataTable.getItems().clear();
+ dataTable.setPlaceholder(new Label(Messages.format("tree.error", e.getMessage())));
+ if (currentQuery.isEmpty()) {
+ Dialogs.showError(getScene() != null ? getScene().getWindow() : null,
+ Messages.format("table.loadError", table), e);
+ } else {
+ // An invalid condition or order is shown below the query
+ showQueryError(Messages.format("data.query.error", e.getMessage()));
+ }
+ });
}
private void show(DataPage page) {
@@ -151,14 +350,14 @@ private void updatePager(boolean busy) {
refreshButton.setDisable(busy);
}
- private static final java.util.regex.Pattern ISO_DATE_TIME = java.util.regex.Pattern.compile(
+ private static final Pattern ISO_DATE_TIME = Pattern.compile(
"^(\\d{4}-\\d{2}-\\d{2})T(\\d{2}:\\d{2}.*)$");
/**
* Text of a cell in one line, ISO date time is shown with a space separator
*/
static String format(String value) {
- java.util.regex.Matcher matcher = ISO_DATE_TIME.matcher(value);
+ Matcher matcher = ISO_DATE_TIME.matcher(value);
String text = matcher.matches() ? matcher.group(1) + " " + matcher.group(2) : value;
return text.replace('\n', ' ').replace('\r', ' ');
}
diff --git a/src/main/java/com/github/sqljam/face/view/DbTypeCells.java b/src/main/java/com/github/sqljam/face/view/DbTypeCells.java
index e73b56e..8b9ebcb 100644
--- a/src/main/java/com/github/sqljam/face/view/DbTypeCells.java
+++ b/src/main/java/com/github/sqljam/face/view/DbTypeCells.java
@@ -15,24 +15,91 @@
*/
package com.github.sqljam.face.view;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
import com.github.sqljam.face.model.ConnectionProfile;
+import com.github.sqljam.impexp.DbCategory;
import com.github.sqljam.impexp.DbType;
+import javafx.scene.Node;
import javafx.scene.control.ComboBox;
+import javafx.scene.control.Label;
import javafx.scene.control.ListCell;
import javafx.scene.control.ListView;
+import javafx.scene.layout.HBox;
+import javafx.scene.layout.VBox;
import javafx.util.StringConverter;
/**
- * @Description: DbTypeCells renders database types and connection profiles in lists and combo boxes
+ * @Description: DbTypeCells renders database types and connection profiles in lists and combo boxes, grouped by
+ * categories (relational and OLAP databases) with a header before the first item of each category
* @Author: Fred Feng
* @Date: 26/03/2023
* @Version 1.0.0
*/
public final class DbTypeCells {
+ /**
+ * Style class of a cell starting a category
+ */
+ static final String CATEGORY_START = "category-start";
+
private DbTypeCells() {
}
+ /**
+ * Database types ordered by category, the order of DbType is kept in a category
+ */
+ public static List getDbTypesByCategory() {
+ return Arrays.stream(DbType.values()).sorted(Comparator.comparing(DbType::getCategory))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * Category of a profile, relational if the type is unknown
+ */
+ public static DbCategory categoryOf(ConnectionProfile profile) {
+ return profile != null && profile.getDbType() != null ? profile.getDbType().getCategory()
+ : DbCategory.RELATIONAL;
+ }
+
+ /**
+ * Whether the item at the index starts a category of the list
+ */
+ static boolean isCategoryStart(List items, int index, Function category) {
+ if (index < 0 || index >= items.size()) {
+ return false;
+ }
+ return index == 0 || category.apply(items.get(index - 1)) != category.apply(items.get(index));
+ }
+
+ /**
+ * Cells of themes have a fixed height, a cell starting a category is higher for its header
+ */
+ static void setCategoryStart(ListCell cell, boolean start) {
+ cell.getStyleClass().remove(CATEGORY_START);
+ // Inline, dialogs have the theme without the stylesheet of the application
+ cell.setStyle(start ? "-fx-cell-size: " + (cell.getListView() != null
+ && cell.getListView().getStyleClass().contains("profile-list") ? "4.6em" : "4em") : null);
+ if (start) {
+ cell.getStyleClass().add(CATEGORY_START);
+ }
+ }
+
+ /**
+ * Content with the category header above it
+ */
+ static Node withHeader(DbCategory category, Node content) {
+ Label header = new Label(category.getDisplayName());
+ header.getStyleClass().add("category-header");
+ header.setStyle("-fx-font-size: 11px; -fx-font-weight: bold; -fx-text-fill: -color-fg-muted;"
+ + " -fx-padding: 4 0 2 0;");
+ return new VBox(2, header, content);
+ }
+
public static ListCell profileCell() {
return new ListCell<>() {
@Override
@@ -65,24 +132,60 @@ public ConnectionProfile fromString(String text) {
});
}
+ /**
+ * Profiles ordered by category have a category header before the first profile of each category
+ */
public static void setupProfileList(ListView listView) {
+ listView.getStyleClass().add("profile-list");
listView.setCellFactory(view -> new ListCell<>() {
@Override
protected void updateItem(ConnectionProfile item, boolean empty) {
super.updateItem(item, empty);
+ setCategoryStart(this, false);
if (empty || item == null) {
setText(null);
setGraphic(null);
+ return;
+ }
+ String text = item.getName() + "\n" + (item.getDbType() != null ? item.getDbType().getDisplayName()
+ : "");
+ if (isCategoryStart(view.getItems(), getIndex(), DbTypeCells::categoryOf)) {
+ setCategoryStart(this, true);
+ setText(null);
+ // The gap of the icon and text of a cell
+ setGraphic(withHeader(categoryOf(item), new HBox(getGraphicTextGap(), Icons.of(Icons.CONNECTION),
+ new Label(text))));
} else {
- setText(item.getName() + "\n" + (item.getDbType() != null ? item.getDbType().getDisplayName()
- : ""));
+ setText(text);
setGraphic(Icons.of(Icons.CONNECTION));
}
}
});
}
+ /**
+ * Types of the combo have a category header before the first type of each category, the selected type is shown
+ * without header
+ */
public static void setupDbTypeCombo(ComboBox combo) {
+ combo.setCellFactory(view -> new ListCell<>() {
+ @Override
+ protected void updateItem(DbType item, boolean empty) {
+ super.updateItem(item, empty);
+ setCategoryStart(this, false);
+ if (empty || item == null) {
+ setText(null);
+ setGraphic(null);
+ } else if (isCategoryStart(view.getItems(), getIndex(), DbType::getCategory)) {
+ setCategoryStart(this, true);
+ setText(null);
+ setGraphic(withHeader(item.getCategory(), new Label(item.getDisplayName())));
+ } else {
+ setText(item.getDisplayName());
+ setGraphic(null);
+ }
+ }
+ });
combo.setConverter(new StringConverter<>() {
@Override
public String toString(DbType dbType) {
diff --git a/src/main/java/com/github/sqljam/face/view/Dialogs.java b/src/main/java/com/github/sqljam/face/view/Dialogs.java
index a329338..fdeee9a 100644
--- a/src/main/java/com/github/sqljam/face/view/Dialogs.java
+++ b/src/main/java/com/github/sqljam/face/view/Dialogs.java
@@ -18,17 +18,21 @@
import java.io.PrintWriter;
import java.io.StringWriter;
import java.util.Optional;
+import java.util.function.Consumer;
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.face.Banner;
+import javafx.geometry.Insets;
import javafx.scene.control.Alert;
import javafx.scene.control.ButtonType;
import javafx.scene.control.Dialog;
+import javafx.scene.control.Hyperlink;
import javafx.scene.control.Label;
import javafx.scene.control.TextArea;
import javafx.scene.image.ImageView;
import javafx.scene.layout.GridPane;
import javafx.scene.layout.Priority;
+import javafx.scene.layout.VBox;
import javafx.stage.Window;
/**
@@ -94,24 +98,61 @@ public static void showInfo(Window owner, String title, String message) {
}
/**
- * About dialog with the logo and the version
+ * About SqlJam with the version, the home page, the source repository, the author and the license
+ *
+ * @param browser opens the links in the browser, null for links without an action
*/
- public static Alert createAbout(Window owner) {
+ public static Alert createAbout(Window owner, Consumer browser) {
Alert alert = new Alert(Alert.AlertType.INFORMATION);
initOwner(alert, owner);
Branding.applyIcons(alert);
alert.setTitle(Messages.get("about.title"));
alert.setHeaderText(Messages.format("about.header", Banner.getVersion()));
- alert.setContentText(Messages.get("about.message"));
ImageView logo = Branding.logoView(96);
logo.setId("aboutLogo");
alert.setGraphic(logo);
- alert.getDialogPane().setPrefWidth(460);
+ alert.getDialogPane().setPrefWidth(500);
+ Label message = new Label(Messages.get("about.message"));
+ message.setWrapText(true);
+ GridPane details = new GridPane();
+ details.setId("aboutDetails");
+ details.setHgap(16);
+ details.setVgap(4);
+ details.addRow(0, aboutLabel("about.version"), new Label(Banner.getVersion()));
+ details.addRow(1, aboutLabel("about.homepage"), aboutLink("aboutHomepage", Branding.HOMEPAGE_URL,
+ Branding.HOMEPAGE_URL, browser));
+ details.addRow(2, aboutLabel("about.repository"), aboutLink("aboutRepository", Branding.REPOSITORY_URL,
+ Branding.REPOSITORY_URL, browser));
+ details.addRow(3, aboutLabel("about.author"), new Label(Branding.AUTHOR));
+ details.addRow(4, aboutLabel("about.email"), aboutLink("aboutEmail", Branding.EMAIL,
+ "mailto:" + Branding.EMAIL, browser));
+ details.addRow(5, aboutLabel("about.license"), new Label(Messages.get("about.licenseName")));
+ alert.getDialogPane().setContent(new VBox(14, message, details));
return alert;
}
- public static void showAbout(Window owner) {
- createAbout(owner).showAndWait();
+ private static Label aboutLabel(String key) {
+ Label label = new Label(Messages.get(key));
+ label.getStyleClass().add("muted");
+ return label;
+ }
+
+ private static Hyperlink aboutLink(String id, String text, String url, Consumer browser) {
+ Hyperlink link = new Hyperlink(text);
+ link.setId(id);
+ link.setPadding(Insets.EMPTY);
+ link.setOnAction(event -> {
+ // Not shown as visited, it is opened again
+ link.setVisited(false);
+ if (browser != null) {
+ browser.accept(url);
+ }
+ });
+ return link;
+ }
+
+ public static void showAbout(Window owner, Consumer browser) {
+ createAbout(owner, browser).showAndWait();
}
public static boolean confirm(Window owner, String title, String message) {
diff --git a/src/main/java/com/github/sqljam/face/view/ExampleActions.java b/src/main/java/com/github/sqljam/face/view/ExampleActions.java
new file mode 100644
index 0000000..41a4aab
--- /dev/null
+++ b/src/main/java/com/github/sqljam/face/view/ExampleActions.java
@@ -0,0 +1,50 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.face.view;
+
+import java.util.function.Consumer;
+
+import com.github.sqljam.face.model.ConnectionProfile;
+import com.github.sqljam.face.service.ExampleDatabase;
+import javafx.stage.Window;
+
+/**
+ * @Description: ExampleActions restores the example database from the login page and the main window, e.g. after the
+ * user removed its data source or its file. An existing example is replaced after a confirmation.
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+public final class ExampleActions {
+
+ private ExampleActions() {
+ }
+
+ /**
+ * Restores the example database in the background
+ *
+ * @param onRestored called with the data source of the example when it is restored
+ */
+ public static void restore(Window owner, AppContext context, Consumer onRestored) {
+ boolean exists = context.getProfileRegistry().findProfile(ExampleDatabase.PROFILE_ID).isPresent();
+ if (exists && !Dialogs.confirm(owner, Messages.get("example.restore.title"),
+ Messages.format("example.restore.message", ExampleDatabase.PROFILE_NAME))) {
+ return;
+ }
+ TaskRunner.run(context::restoreExample, onRestored, e -> Dialogs.showError(owner,
+ Messages.get("example.restore.error"), e));
+ }
+}
diff --git a/src/main/java/com/github/sqljam/face/view/ExportDialog.java b/src/main/java/com/github/sqljam/face/view/ExportDialog.java
index ae376fd..c5ac061 100644
--- a/src/main/java/com/github/sqljam/face/view/ExportDialog.java
+++ b/src/main/java/com/github/sqljam/face/view/ExportDialog.java
@@ -20,6 +20,7 @@
import java.util.EnumMap;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;
@@ -28,10 +29,15 @@
import com.github.sqljam.face.model.TransferRequest;
import com.github.sqljam.face.service.DatabaseSession;
import com.github.sqljam.impexp.DataFileStrategy;
+import com.github.sqljam.impexp.DataFormat;
import com.github.sqljam.impexp.DbType;
+import com.github.sqljam.impexp.Dialect;
import com.github.sqljam.impexp.ExportMode;
import com.github.sqljam.impexp.Exporter;
import com.github.sqljam.impexp.IdentifierCase;
+import com.github.sqljam.impexp.ParquetExporter;
+import com.github.sqljam.impexp.TableQuery;
+import javafx.beans.InvalidationListener;
import javafx.beans.property.BooleanProperty;
import javafx.beans.property.SimpleBooleanProperty;
import javafx.collections.FXCollections;
@@ -51,6 +57,7 @@
import javafx.scene.control.RadioButton;
import javafx.scene.control.ScrollPane;
import javafx.scene.control.Spinner;
+import javafx.scene.control.SpinnerValueFactory;
import javafx.scene.control.TextField;
import javafx.scene.control.ToggleGroup;
import javafx.scene.control.cell.CheckBoxListCell;
@@ -76,6 +83,7 @@ public class ExportDialog extends Dialog {
static {
VERSION_PRESETS.put(DbType.MYSQL, List.of("", "5.6", "5.7", "8.0"));
+ VERSION_PRESETS.put(DbType.MARIADB, List.of("", "10.6", "10.11", "11.4", "11.8"));
VERSION_PRESETS.put(DbType.POSTGRESQL, List.of("", "9.6", "10", "12", "16"));
VERSION_PRESETS.put(DbType.ORACLE, List.of("", "11.2", "12.1", "19", "23"));
VERSION_PRESETS.put(DbType.SQLSERVER, List.of("", "2008", "2012", "2016", "2019", "2022"));
@@ -119,10 +127,18 @@ public class ExportDialog extends Dialog {
private final RadioButton perTableRadio = new RadioButton(Messages.get("export.script.perTable"));
private final Spinner maxFileSizeSpinner = new Spinner<>(0, 100000, 10, 10);
private final CheckBox lobSeparatedCheck = new CheckBox(Messages.get("export.script.lobSeparated"));
+ private final ToggleGroup formatGroup = new ToggleGroup();
+ private final RadioButton sqlFormatRadio = new RadioButton(Messages.get("export.format.sql"));
+ private final RadioButton parquetFormatRadio = new RadioButton(Messages.get("export.format.parquet"));
+ private final ComboBox compressionCombo = new ComboBox<>();
private final Label layoutHint = new Label();
private final ComboBox targetCombo = new ComboBox<>();
private final ComboBox targetCatalogCombo = new ComboBox<>();
private final ComboBox targetSchemaCombo = new ComboBox<>();
+ private final TextField tableNamePatternField = new TextField();
+ private final Label queryLabel = new Label();
+ private TableQuery tableQuery;
+ private String queryTableName;
private final CheckBox createSchemaCheck = new CheckBox(Messages.get("export.database.createSchema"));
private final Label errorLabel = new Label();
private Button startButton;
@@ -142,7 +158,16 @@ private static class TableChoice {
}
public ExportDialog(Window owner, AppContext context, DbNode node) {
+ this(owner, context, node, null);
+ }
+
+ /**
+ * Export of a table narrowed by a query of the data viewer: selected columns, condition and order
+ */
+ public ExportDialog(Window owner, AppContext context, DbNode node, TableQuery query) {
this.context = context;
+ this.tableQuery = query != null && !query.isEmpty() && node != null && node.getTable() != null ? query : null;
+ this.queryTableName = tableQuery != null ? node.getTable().getName() : null;
Dialogs.initOwner(this, owner);
Branding.applyIcons(this);
setTitle(Messages.get("export.title"));
@@ -166,10 +191,14 @@ public ExportDialog(Window owner, AppContext context, DbNode node) {
perTableRadio.setId("perTableRadio");
maxFileSizeSpinner.setId("maxFileSizeSpinner");
lobSeparatedCheck.setId("lobSeparatedCheck");
+ sqlFormatRadio.setId("sqlFormatRadio");
+ parquetFormatRadio.setId("parquetFormatRadio");
+ compressionCombo.setId("compressionCombo");
layoutHint.setId("layoutHint");
targetCombo.setId("targetCombo");
targetCatalogCombo.setId("targetCatalogCombo");
targetSchemaCombo.setId("targetSchemaCombo");
+ tableNamePatternField.setId("tableNamePatternField");
createSchemaCheck.setId("createSchemaCheck");
errorLabel.setId("errorLabel");
recreateCheck.setId("recreateCheck");
@@ -207,6 +236,11 @@ public ExportDialog(Window owner, AppContext context, DbNode node) {
setResultConverter(buttonType -> buttonType == ButtonType.OK ? buildRequest() : null);
registerValidation();
preselect(node);
+ if (tableQuery != null) {
+ queryLabel.setText(Messages.format("export.query", queryTableName, tableQuery));
+ queryLabel.setVisible(true);
+ queryLabel.setManaged(true);
+ }
updateValidation();
}
@@ -214,11 +248,17 @@ public ExportDialog(Window owner, AppContext context, DbNode node) {
* Start is enabled only when the inputs are valid, the reason is shown below the form
*/
private void registerValidation() {
- javafx.beans.InvalidationListener listener = observable -> updateValidation();
+ InvalidationListener listener = observable -> updateValidation();
sourceCombo.valueProperty().addListener(listener);
targetGroup.selectedToggleProperty().addListener(listener);
directoryField.textProperty().addListener(listener);
targetCombo.valueProperty().addListener(listener);
+ targetCatalogCombo.valueProperty().addListener(listener);
+ targetSchemaCombo.valueProperty().addListener(listener);
+ targetSchemaCombo.getEditor().textProperty().addListener(listener);
+ tableNamePatternField.textProperty().addListener(listener);
+ catalogCombo.valueProperty().addListener(listener);
+ schemaCombo.valueProperty().addListener(listener);
scriptVersionCombo.getEditor().textProperty().addListener(listener);
scriptVersionCombo.valueProperty().addListener(listener);
pageSizeSpinner.getEditor().textProperty().addListener(listener);
@@ -305,6 +345,13 @@ public TableChoice fromString(String text) {
grid.addRow(1, new Label(Messages.get("export.source.catalog")), catalogCombo);
grid.addRow(2, new Label(Messages.get("export.source.schema")), schemaCombo);
VBox box = new VBox(8, grid, new Label(Messages.get("export.tables.hint")), filterBox, tableList);
+ // Rows of the table narrowed in the data viewer
+ queryLabel.setId("queryLabel");
+ queryLabel.getStyleClass().add("muted");
+ queryLabel.setWrapText(true);
+ queryLabel.setVisible(false);
+ queryLabel.setManaged(false);
+ box.getChildren().add(queryLabel);
return box;
}
@@ -373,7 +420,7 @@ private Node createTargetPane() {
browseButton.setOnAction(event -> chooseDirectory());
HBox directoryBox = new HBox(8, directoryField, browseButton);
HBox.setHgrow(directoryField, Priority.ALWAYS);
- scriptDbTypeCombo.getItems().addAll(DbType.values());
+ scriptDbTypeCombo.getItems().addAll(DbTypeCells.getDbTypesByCategory());
DbTypeCells.setupDbTypeCombo(scriptDbTypeCombo);
scriptDbTypeCombo.valueProperty().addListener((obs, oldType, type) -> {
scriptVersionCombo.getItems().setAll(VERSION_PRESETS.getOrDefault(type, List.of("")));
@@ -389,6 +436,18 @@ private Node createTargetPane() {
long maxFileSize = context.getSettings().getMaxDataFileSize();
maxFileSizeSpinner.getValueFactory().setValue(maxFileSize > 0 ? (int) Math.max(1, maxFileSize / MB) : 10);
lobSeparatedCheck.setSelected(true);
+ sqlFormatRadio.setToggleGroup(formatGroup);
+ parquetFormatRadio.setToggleGroup(formatGroup);
+ sqlFormatRadio.setSelected(true);
+ compressionCombo.getItems().setAll(ParquetExporter.COMPRESSIONS);
+ compressionCombo.setValue(ParquetExporter.DEFAULT_COMPRESSION);
+ // Options of sql scripts do not apply to Parquet files
+ compressionCombo.disableProperty().bind(parquetFormatRadio.selectedProperty().not());
+ singleFileRadio.disableProperty().bind(parquetFormatRadio.selectedProperty());
+ perTableRadio.disableProperty().bind(parquetFormatRadio.selectedProperty());
+ maxFileSizeSpinner.disableProperty().bind(parquetFormatRadio.selectedProperty());
+ lobSeparatedCheck.disableProperty().bind(parquetFormatRadio.selectedProperty());
+ formatGroup.selectedToggleProperty().addListener((obs, oldToggle, toggle) -> updateLayoutHint());
layoutHint.getStyleClass().addAll("muted", "mono");
layoutHint.setWrapText(true);
strategyGroup.selectedToggleProperty().addListener((obs, oldToggle, toggle) -> updateLayoutHint());
@@ -401,6 +460,9 @@ private Node createTargetPane() {
scriptGrid.addRow(row++, new Label(Messages.get("export.script.dbType")), scriptDbTypeCombo);
scriptGrid.addRow(row++, new Label(Messages.get("export.script.version")), scriptVersionCombo);
scriptGrid.addRow(row++, new Label(Messages.get("export.script.schema")), scriptSchemaField);
+ scriptGrid.addRow(row++, new Label(Messages.get("export.format")), new HBox(16, sqlFormatRadio,
+ parquetFormatRadio));
+ scriptGrid.addRow(row++, new Label(Messages.get("export.format.compression")), compressionCombo);
scriptGrid.addRow(row++, new Label(Messages.get("export.script.strategy")), new HBox(16, singleFileRadio,
perTableRadio));
scriptGrid.addRow(row++, new Label(Messages.get("export.script.maxFileSize")), new HBox(8,
@@ -426,10 +488,20 @@ private Node createTargetPane() {
}
});
GridPane databaseGrid = grid();
- databaseGrid.addRow(0, new Label(Messages.get("export.database.connection")), targetCombo);
+ databaseGrid.addRow(0, new Label(Messages.get("export.database.connection")), ConnectionDialog.targetField(
+ targetCombo, context, profile -> {
+ targetCombo.getItems().setAll(context.getProfileRegistry().getProfiles());
+ ConnectionDialog.selectProfile(targetCombo, profile);
+ }));
databaseGrid.addRow(1, new Label(Messages.get("export.database.catalog")), targetCatalogCombo);
databaseGrid.addRow(2, new Label(Messages.get("export.database.schema")), targetSchemaCombo);
databaseGrid.add(createSchemaCheck, 1, 3);
+ tableNamePatternField.setPromptText(Dialect.TABLE_PLACEHOLDER);
+ Label tableNameHint = new Label(Messages.get("export.database.tableName.hint"));
+ tableNameHint.getStyleClass().add("muted");
+ tableNameHint.setWrapText(true);
+ databaseGrid.addRow(4, new Label(Messages.get("export.database.tableName")), tableNamePatternField);
+ databaseGrid.add(tableNameHint, 1, 5);
VBox databasePane = new VBox(databaseGrid);
databasePane.getStyleClass().add("target-pane");
@@ -441,6 +513,10 @@ private Node createTargetPane() {
}
private void updateLayoutHint() {
+ if (parquetFormatRadio.isSelected()) {
+ layoutHint.setText("schema.sql\ndata/.parquet\nconstraints.sql\nmanifest.json");
+ return;
+ }
boolean perTable = perTableRadio.isSelected();
boolean split = maxFileSizeSpinner.getValue() != null && maxFileSizeSpinner.getValue() > 0;
StringBuilder hint = new StringBuilder();
@@ -647,8 +723,13 @@ private void loadTargetSchemas(ConnectionProfile profile, String catalog) {
}, this::showError);
}
+ /**
+ * Errors of loading after the dialog is closed are not shown
+ */
private void showError(Throwable e) {
- Dialogs.showError(getDialogPane().getScene().getWindow(), Messages.get("export.loadError"), e);
+ if (isShowing()) {
+ Dialogs.showError(getDialogPane().getScene().getWindow(), Messages.get("export.loadError"), e);
+ }
}
String validate() {
@@ -679,17 +760,53 @@ String validate() {
}
} else if (targetCombo.getValue() == null) {
return Messages.get("export.error.target");
+ } else {
+ return validateTableNames();
}
return null;
}
+ /**
+ * Tables are not imported into themselves, tables copied in the same schema need another name. A name without
+ * {table} is the name of a single table.
+ */
+ private String validateTableNames() {
+ String pattern = StringUtils.trimToEmpty(tableNamePatternField.getText());
+ boolean renamed = !pattern.isEmpty() && !Dialect.TABLE_PLACEHOLDER.equals(pattern);
+ if (renamed && !pattern.contains(Dialect.TABLE_PLACEHOLDER) && getSelectedTableCount() != 1) {
+ return Messages.format("export.error.tableName", Dialect.TABLE_PLACEHOLDER);
+ }
+ ConnectionProfile source = sourceCombo.getValue();
+ ConnectionProfile target = targetCombo.getValue();
+ if (!renamed && source != null && target != null && Objects.equals(source.getId(), target.getId())
+ && isSameName(catalogCombo.isDisabled() ? null : catalogCombo.getValue(),
+ targetCatalogCombo.isDisabled() ? null : targetCatalogCombo.getValue())
+ && isSameName(schemaCombo.isDisabled() ? null : schemaCombo.getValue(),
+ targetSchemaCombo.isDisabled() ? null : targetSchemaCombo.getEditor().getText())) {
+ return Messages.format("export.error.sameTables", Dialect.TABLE_PLACEHOLDER);
+ }
+ return null;
+ }
+
+ /**
+ * A target name which is not given is the source name
+ */
+ private static boolean isSameName(String sourceName, String targetName) {
+ return StringUtils.isBlank(targetName) || StringUtils.isBlank(sourceName)
+ || sourceName.trim().equalsIgnoreCase(targetName.trim());
+ }
+
+ private int getSelectedTableCount() {
+ return (int) tableChoices.stream().filter(choice -> choice.selected.get()).count();
+ }
+
private static int getMin(Spinner spinner) {
- return ((javafx.scene.control.SpinnerValueFactory.IntegerSpinnerValueFactory) spinner.getValueFactory())
+ return ((SpinnerValueFactory.IntegerSpinnerValueFactory) spinner.getValueFactory())
.getMin();
}
private static int getMax(Spinner spinner) {
- return ((javafx.scene.control.SpinnerValueFactory.IntegerSpinnerValueFactory) spinner.getValueFactory())
+ return ((SpinnerValueFactory.IntegerSpinnerValueFactory) spinner.getValueFactory())
.getMax();
}
@@ -751,6 +868,8 @@ TransferRequest buildRequest() {
: DataFileStrategy.SINGLE_FILE);
request.setMaxFileSize(maxFileSizeSpinner.getValue() * MB);
request.setLobSeparated(lobSeparatedCheck.isSelected());
+ request.setDataFormat(parquetFormatRadio.isSelected() ? DataFormat.PARQUET : DataFormat.SQL);
+ request.setCompression(compressionCombo.getValue());
context.getSettings().setLastExportDirectory(directory.getAbsolutePath());
context.getSettings().setMaxDataFileSize(maxFileSizeSpinner.getValue() * MB);
} else {
@@ -760,6 +879,11 @@ TransferRequest buildRequest() {
request.setTargetSchema(targetSchemaCombo.isDisabled() ? null : StringUtils.trimToNull(
targetSchemaCombo.getEditor().getText()));
request.setTargetSchemaCreated(createSchemaCheck.isSelected());
+ request.setTableNamePattern(StringUtils.trimToNull(tableNamePatternField.getText()));
+ }
+ // The query of the data viewer narrows its table
+ if (tableQuery != null && selected.contains(queryTableName)) {
+ request.getTableQueries().put(queryTableName, tableQuery);
}
context.getSettings().setTransferPageSize(pageSizeSpinner.getValue());
context.saveSettings();
diff --git a/src/main/java/com/github/sqljam/face/view/Icons.java b/src/main/java/com/github/sqljam/face/view/Icons.java
index 647ecb7..bdc8e67 100644
--- a/src/main/java/com/github/sqljam/face/view/Icons.java
+++ b/src/main/java/com/github/sqljam/face/view/Icons.java
@@ -51,6 +51,9 @@ public final class Icons {
public static final String LAST = "fth-chevrons-right";
public static final String CONNECT = "fth-link";
public static final String OPEN = "fth-external-link";
+ public static final String GITHUB = "fth-github";
+ public static final String DISCONNECT = "fth-power";
+ public static final String EXIT = "fth-log-out";
private Icons() {
}
diff --git a/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java b/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java
index e5bd329..f6ab515 100644
--- a/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java
+++ b/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java
@@ -17,13 +17,16 @@
import java.io.File;
import java.util.List;
+import java.util.Locale;
import java.util.Objects;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.face.model.ConnectionProfile;
+import com.github.sqljam.impexp.DataFormat;
import com.github.sqljam.impexp.DbType;
import com.github.sqljam.impexp.ExportManifest;
+import com.github.sqljam.impexp.ParquetExporter;
import javafx.beans.property.ReadOnlyObjectWrapper;
import javafx.event.ActionEvent;
import javafx.geometry.Insets;
@@ -61,6 +64,8 @@ public class ImportPackageDialog extends Dialog tableView = new TableView<>();
private final VBox summaryPane;
private final ComboBox targetCombo = new ComboBox<>();
@@ -110,6 +115,8 @@ public ImportPackageDialog(Window owner, AppContext context, DbNode node) {
sourceValue.setId("sourceValue");
targetValue.setId("targetValue");
filesValue.setId("filesValue");
+ formatValue.setId("formatValue");
+ viewButton.setId("viewPackageButton");
tableView.setId("packageTables");
targetCombo.setId("targetCombo");
catalogCombo.setId("catalogCombo");
@@ -124,7 +131,14 @@ public ImportPackageDialog(Window owner, AppContext context, DbNode node) {
});
Button browseButton = new Button(null, Icons.of(Icons.FOLDER));
browseButton.setOnAction(event -> chooseDirectory());
- HBox directoryBox = new HBox(8, directoryField, browseButton);
+ // Files of the package are listed and previewed
+ viewButton.setOnAction(event -> {
+ File directory = new File(directoryField.getText().trim());
+ if (directory.isDirectory()) {
+ new PackageViewer(getDialogPane().getScene().getWindow(), context, directory).show();
+ }
+ });
+ HBox directoryBox = new HBox(8, directoryField, browseButton, viewButton);
HBox.setHgrow(directoryField, Priority.ALWAYS);
GridPane summaryGrid = grid();
@@ -133,6 +147,7 @@ public ImportPackageDialog(Window owner, AppContext context, DbNode node) {
summaryGrid.addRow(2, new Label(Messages.get("import.manifest.source")), sourceValue);
summaryGrid.addRow(3, new Label(Messages.get("import.manifest.target")), targetValue);
summaryGrid.addRow(4, new Label(Messages.get("import.manifest.files")), filesValue);
+ summaryGrid.addRow(5, new Label(Messages.get("export.format")), formatValue);
setupTableView();
Label summaryTitle = new Label(Messages.get("import.section.package"));
summaryTitle.getStyleClass().add("section-title");
@@ -157,7 +172,12 @@ public ImportPackageDialog(Window owner, AppContext context, DbNode node) {
}
});
GridPane targetGrid = grid();
- targetGrid.addRow(0, new Label(Messages.get("export.database.connection")), targetCombo);
+ targetGrid.addRow(0, new Label(Messages.get("export.database.connection")), ConnectionDialog.targetField(
+ targetCombo, context, profile -> {
+ // A data source of another type than the package is not listed
+ filterTargets();
+ ConnectionDialog.selectProfile(targetCombo, profile);
+ }));
targetGrid.addRow(1, new Label(Messages.get("export.database.catalog")), catalogCombo);
targetGrid.addRow(2, new Label(Messages.get("export.database.schema")), schemaCombo);
targetGrid.add(stopOnErrorCheck, 1, 3);
@@ -273,10 +293,16 @@ private void loadManifest() {
return;
}
statusValue.setText(Messages.get("tree.loading"));
- TaskRunner.run(() -> new Object[]{ExportManifest.read(directory), isScriptDirectory(directory)}, result -> {
- directoryValid = (Boolean) result[1] || result[0] != null;
+ TaskRunner.run(() -> new Object[]{ExportManifest.read(directory), isScriptDirectory(directory),
+ isParquetDirectory(directory)}, result -> {
+ // Parquet files without manifest are not sql scripts, even with schema.sql
+ directoryValid = result[0] != null || (Boolean) result[1] && !(Boolean) result[2];
showSummary((ExportManifest) result[0], directoryValid);
validateTarget();
+ // Parquet files of other tools have no manifest, they are loaded by Import Parquet Files
+ if (!directoryValid && (Boolean) result[2]) {
+ warningLabel.setText(Messages.get("import.warning.parquetFiles"));
+ }
}, e -> {
directoryValid = false;
showSummary(null, false);
@@ -289,6 +315,19 @@ static boolean isScriptDirectory(File directory) {
|| new File(directory, "data").isDirectory();
}
+ /**
+ * Directory with Parquet files (directly or in data/) but without manifest.json
+ */
+ static boolean isParquetDirectory(File directory) {
+ return hasParquetFiles(directory) || hasParquetFiles(new File(directory, "data"));
+ }
+
+ private static boolean hasParquetFiles(File directory) {
+ File[] files = directory.listFiles((parent, name) -> name.toLowerCase(Locale.ENGLISH)
+ .endsWith(ParquetExporter.PARQUET_EXTENSION));
+ return files != null && files.length > 0;
+ }
+
private void showSummary(ExportManifest manifest, boolean valid) {
this.manifest = manifest;
tableView.getItems().clear();
@@ -299,6 +338,7 @@ private void showSummary(ExportManifest manifest, boolean valid) {
sourceValue.setText("-");
targetValue.setText("-");
filesValue.setText("-");
+ formatValue.setText(valid ? Messages.get("export.format.sql") : "-");
filterTargets();
return;
}
@@ -312,6 +352,8 @@ private void showSummary(ExportManifest manifest, boolean valid) {
long totalSize = manifest.getFiles().stream().mapToLong(ExportManifest.FileEntry::getSize).sum();
filesValue.setText(Messages.format("import.manifest.filesValue", manifest.getFiles().size(),
FileUtils.byteCountToDisplaySize(totalSize)));
+ formatValue.setText(Messages.get(manifest.getDataFormat() == DataFormat.PARQUET ? "export.format.parquet"
+ : "export.format.sql"));
tableView.getItems().setAll(manifest.getTables());
filterTargets();
}
@@ -412,7 +454,16 @@ private void loadCatalogs(ConnectionProfile profile) {
: catalogs.contains(profile.getDatabase()) ? profile.getDatabase() : catalogs.get(0);
catalogCombo.setValue(preferred);
schemaCombo.setDisable(!profile.getDbType().isSchemaSupported());
- }, e -> Dialogs.showError(getDialogPane().getScene().getWindow(), Messages.get("export.loadError"), e));
+ }, this::showLoadError);
+ }
+
+ /**
+ * Errors of loading after the dialog is closed are not shown
+ */
+ private void showLoadError(Throwable e) {
+ if (isShowing()) {
+ Dialogs.showError(getDialogPane().getScene().getWindow(), Messages.get("export.loadError"), e);
+ }
}
private void loadSchemas(ConnectionProfile profile, String catalog) {
@@ -424,7 +475,7 @@ private void loadSchemas(ConnectionProfile profile, String catalog) {
if (StringUtils.isNotBlank(preferred)) {
schemaCombo.setValue(preferred);
}
- }, e -> Dialogs.showError(getDialogPane().getScene().getWindow(), Messages.get("export.loadError"), e));
+ }, this::showLoadError);
}
ImportRequest buildRequest() {
diff --git a/src/main/java/com/github/sqljam/face/view/ImportParquetDialog.java b/src/main/java/com/github/sqljam/face/view/ImportParquetDialog.java
new file mode 100644
index 0000000..ffa5644
--- /dev/null
+++ b/src/main/java/com/github/sqljam/face/view/ImportParquetDialog.java
@@ -0,0 +1,408 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.face.view;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.commons.io.FilenameUtils;
+import org.apache.commons.lang3.StringUtils;
+import com.github.sqljam.face.model.ConnectionProfile;
+import com.github.sqljam.face.service.TransferService;
+import com.github.sqljam.impexp.ParquetExporter;
+import com.github.sqljam.impexp.ParquetImporter;
+import javafx.beans.property.ReadOnlyObjectWrapper;
+import javafx.event.ActionEvent;
+import javafx.geometry.Insets;
+import javafx.scene.control.Button;
+import javafx.scene.control.ButtonType;
+import javafx.scene.control.ComboBox;
+import javafx.scene.control.Dialog;
+import javafx.scene.control.Label;
+import javafx.scene.control.ListCell;
+import javafx.scene.control.ListView;
+import javafx.scene.control.RadioButton;
+import javafx.scene.control.SelectionMode;
+import javafx.scene.control.TableColumn;
+import javafx.scene.control.TableView;
+import javafx.scene.control.TextField;
+import javafx.scene.control.ToggleGroup;
+import javafx.scene.control.Tooltip;
+import javafx.scene.layout.ColumnConstraints;
+import javafx.scene.layout.GridPane;
+import javafx.scene.layout.HBox;
+import javafx.scene.layout.Priority;
+import javafx.scene.layout.VBox;
+import javafx.stage.FileChooser;
+import javafx.stage.Window;
+import lombok.Getter;
+
+/**
+ * @Description: ImportParquetDialog loads Parquet files (of SqlJam or other tools) into a table of a database. Files
+ * of one table are chosen together, columns and the first rows are previewed, the table is created
+ * (or replaced) or the rows are appended to an existing table.
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+public class ImportParquetDialog extends Dialog {
+
+ public static final int PREVIEW_ROWS = 50;
+
+ private final AppContext context;
+ private final ListView fileList = new ListView<>();
+ private final Label previewLabel = new Label();
+ private final TableView> previewTable = new TableView<>();
+ private final TextField tableField = new TextField();
+ private final RadioButton createRadio = new RadioButton(Messages.get("parquet.mode.create"));
+ private final RadioButton appendRadio = new RadioButton(Messages.get("parquet.mode.append"));
+ private final ComboBox targetCombo = new ComboBox<>();
+ private final ComboBox catalogCombo = new ComboBox<>();
+ private final ComboBox schemaCombo = new ComboBox<>();
+ private final Label warningLabel = new Label();
+ private final Button startButton;
+ private boolean filesValid;
+ private int previewVersion;
+
+ /**
+ * Load of Parquet files into a table of the target connection
+ */
+ @Getter
+ public static class ParquetRequest {
+
+ private final List files;
+ private final String tableName;
+ private final ParquetImporter.Mode mode;
+ private final ConnectionProfile target;
+ private final String catalog;
+ private final String schema;
+
+ ParquetRequest(List files, String tableName, ParquetImporter.Mode mode, ConnectionProfile target,
+ String catalog, String schema) {
+ this.files = files;
+ this.tableName = tableName;
+ this.mode = mode;
+ this.target = target;
+ this.catalog = catalog;
+ this.schema = schema;
+ }
+ }
+
+ public ImportParquetDialog(Window owner, AppContext context, DbNode node) {
+ this.context = context;
+ Dialogs.initOwner(this, owner);
+ Branding.applyIcons(this);
+ setTitle(Messages.get("parquet.title"));
+ setHeaderText(Messages.get("parquet.header"));
+ setResizable(true);
+
+ fileList.setId("parquetFiles");
+ previewLabel.setId("parquetPreviewLabel");
+ previewTable.setId("parquetPreview");
+ tableField.setId("tableField");
+ createRadio.setId("createRadio");
+ appendRadio.setId("appendRadio");
+ targetCombo.setId("targetCombo");
+ catalogCombo.setId("catalogCombo");
+ schemaCombo.setId("schemaCombo");
+ warningLabel.setId("warningLabel");
+
+ fileList.setPrefHeight(110);
+ // The name of a file with its folder, the whole path is shown by the tooltip
+ fileList.setCellFactory(view -> new ListCell<>() {
+ @Override
+ protected void updateItem(File file, boolean empty) {
+ super.updateItem(file, empty);
+ if (empty || file == null) {
+ setText(null);
+ setGraphic(null);
+ setTooltip(null);
+ return;
+ }
+ Label folder = new Label(file.getParent());
+ folder.getStyleClass().add("muted");
+ setText(null);
+ setGraphic(new HBox(8, Icons.of(Icons.TABLE), new Label(file.getName()), folder));
+ setTooltip(new Tooltip(file.getAbsolutePath()));
+ }
+ });
+ fileList.setPlaceholder(new Label(Messages.get("parquet.files.empty")));
+ fileList.getSelectionModel().setSelectionMode(SelectionMode.MULTIPLE);
+ Button addButton = new Button(Messages.get("parquet.files.add"), Icons.of(Icons.ADD));
+ addButton.setId("addFilesButton");
+ addButton.setOnAction(event -> chooseFiles());
+ Button removeButton = new Button(Messages.get("parquet.files.remove"), Icons.of(Icons.DELETE));
+ removeButton.setId("removeFilesButton");
+ removeButton.disableProperty().bind(fileList.getSelectionModel().selectedItemProperty().isNull());
+ removeButton.setOnAction(event -> removeFiles(new ArrayList<>(fileList.getSelectionModel()
+ .getSelectedItems())));
+ VBox fileButtons = new VBox(8, addButton, removeButton);
+ addButton.setMaxWidth(Double.MAX_VALUE);
+ removeButton.setMaxWidth(Double.MAX_VALUE);
+ HBox fileBox = new HBox(8, fileList, fileButtons);
+ HBox.setHgrow(fileList, Priority.ALWAYS);
+
+ previewLabel.getStyleClass().add("muted");
+ previewTable.setPrefHeight(200);
+ previewTable.setPlaceholder(new Label(Messages.get("package.noRows")));
+ Label filesTitle = new Label(Messages.get("parquet.section.files"));
+ filesTitle.getStyleClass().add("section-title");
+ VBox filesPane = new VBox(8, filesTitle, fileBox, previewLabel, previewTable);
+ filesPane.getStyleClass().add("form-section");
+
+ ToggleGroup modeGroup = new ToggleGroup();
+ createRadio.setToggleGroup(modeGroup);
+ appendRadio.setToggleGroup(modeGroup);
+ createRadio.setSelected(true);
+ DbTypeCells.setupProfileCombo(targetCombo);
+ targetCombo.setMaxWidth(Double.MAX_VALUE);
+ catalogCombo.setMaxWidth(Double.MAX_VALUE);
+ schemaCombo.setMaxWidth(Double.MAX_VALUE);
+ schemaCombo.setEditable(true);
+ schemaCombo.setPromptText(Messages.get("import.schema.prompt"));
+ tableField.setPromptText(Messages.get("parquet.table.prompt"));
+ tableField.textProperty().addListener((obs, oldText, text) -> validate());
+ targetCombo.valueProperty().addListener((obs, oldProfile, profile) -> {
+ loadCatalogs(profile);
+ validate();
+ });
+ catalogCombo.valueProperty().addListener((obs, oldCatalog, catalog) -> {
+ ConnectionProfile profile = targetCombo.getValue();
+ if (catalog != null && profile != null && profile.getDbType().isSchemaSupported()) {
+ loadSchemas(profile, catalog);
+ }
+ });
+ GridPane targetGrid = grid();
+ targetGrid.addRow(0, new Label(Messages.get("export.database.connection")), ConnectionDialog.targetField(
+ targetCombo, context, profile -> {
+ targetCombo.getItems().setAll(context.getProfileRegistry().getProfiles());
+ ConnectionDialog.selectProfile(targetCombo, profile);
+ }));
+ targetGrid.addRow(1, new Label(Messages.get("export.database.catalog")), catalogCombo);
+ targetGrid.addRow(2, new Label(Messages.get("export.database.schema")), schemaCombo);
+ targetGrid.addRow(3, new Label(Messages.get("parquet.table")), tableField);
+ targetGrid.addRow(4, new Label(Messages.get("parquet.mode")), new HBox(16, createRadio, appendRadio));
+ Label targetTitle = new Label(Messages.get("import.section.target"));
+ targetTitle.getStyleClass().add("section-title");
+ VBox targetPane = new VBox(8, targetTitle, targetGrid);
+ targetPane.getStyleClass().add("form-section");
+
+ warningLabel.getStyleClass().add("status-error");
+ warningLabel.setWrapText(true);
+ VBox content = new VBox(14, filesPane, targetPane, warningLabel);
+ content.setPadding(new Insets(4));
+ content.setPrefWidth(760);
+ getDialogPane().setContent(content);
+ getDialogPane().getButtonTypes().addAll(ButtonType.OK, ButtonType.CANCEL);
+ startButton = (Button) getDialogPane().lookupButton(ButtonType.OK);
+ startButton.setId("startButton");
+ startButton.setText(Messages.get("import.start"));
+ startButton.setGraphic(Icons.of(Icons.START));
+ startButton.addEventFilter(ActionEvent.ACTION, event -> {
+ if (!isReady()) {
+ event.consume();
+ }
+ });
+ setResultConverter(buttonType -> buttonType == ButtonType.OK ? buildRequest() : null);
+
+ targetCombo.getItems().setAll(context.getProfileRegistry().getProfiles());
+ ConnectionProfile selected = node != null && node.getProfile() != null ? targetCombo.getItems().stream()
+ .filter(profile -> profile.getId().equals(node.getProfile().getId())).findFirst().orElse(null) : null;
+ targetCombo.setValue(selected != null ? selected : targetCombo.getItems().isEmpty() ? null
+ : targetCombo.getItems().get(0));
+ showPreview(null);
+ validate();
+ }
+
+ private static GridPane grid() {
+ GridPane grid = new GridPane();
+ grid.setHgap(10);
+ grid.setVgap(8);
+ ColumnConstraints labelColumn = new ColumnConstraints();
+ labelColumn.setMinWidth(150);
+ ColumnConstraints fieldColumn = new ColumnConstraints();
+ fieldColumn.setHgrow(Priority.ALWAYS);
+ grid.getColumnConstraints().addAll(labelColumn, fieldColumn);
+ return grid;
+ }
+
+ private void chooseFiles() {
+ FileChooser chooser = new FileChooser();
+ chooser.setTitle(Messages.get("parquet.files.add"));
+ chooser.getExtensionFilters().add(new FileChooser.ExtensionFilter(Messages.get("parquet.files.filter"),
+ "*" + ParquetExporter.PARQUET_EXTENSION));
+ String lastDirectory = context.getSettings().getLastImportDirectory();
+ if (StringUtils.isNotBlank(lastDirectory) && new File(lastDirectory).isDirectory()) {
+ chooser.setInitialDirectory(new File(lastDirectory));
+ }
+ List files = chooser.showOpenMultipleDialog(getDialogPane().getScene().getWindow());
+ if (files != null && !files.isEmpty()) {
+ context.getSettings().setLastImportDirectory(files.get(0).getParentFile().getAbsolutePath());
+ context.saveSettings();
+ addFiles(files);
+ }
+ }
+
+ /**
+ * Adds Parquet files of a table, the table name is the base name of the first file by default
+ */
+ public void addFiles(List files) {
+ for (File file : files) {
+ if (!fileList.getItems().contains(file)) {
+ fileList.getItems().add(file);
+ }
+ }
+ if (StringUtils.isBlank(tableField.getText()) && !fileList.getItems().isEmpty()) {
+ tableField.setText(getTableName(fileList.getItems().get(0)));
+ }
+ loadPreview();
+ }
+
+ void removeFiles(List files) {
+ fileList.getItems().removeAll(files);
+ loadPreview();
+ }
+
+ /**
+ * Base name of a Parquet file as an identifier: other characters are replaced by underscores
+ */
+ static String getTableName(File file) {
+ String name = FilenameUtils.getBaseName(file.getName());
+ // Files of a schema qualified table are named schema.table.parquet
+ if (name.contains(".")) {
+ name = name.substring(name.lastIndexOf('.') + 1);
+ }
+ name = name.replaceAll("[^\\p{L}\\p{N}_]", "_");
+ return name.isEmpty() || Character.isDigit(name.charAt(0)) ? "t_" + name : name;
+ }
+
+ private void loadPreview() {
+ List files = new ArrayList<>(fileList.getItems());
+ int version = ++previewVersion;
+ filesValid = false;
+ if (files.isEmpty()) {
+ showPreview(null);
+ validate();
+ return;
+ }
+ previewLabel.setText(Messages.get("package.loading"));
+ previewTable.getColumns().clear();
+ previewTable.getItems().clear();
+ validate();
+ TaskRunner.run(() -> context.getTransferService().previewParquet(files, PREVIEW_ROWS), preview -> {
+ if (version == previewVersion) {
+ filesValid = true;
+ showPreview(preview);
+ validate();
+ }
+ }, e -> {
+ if (version == previewVersion) {
+ showPreview(null);
+ warningLabel.setText(Messages.format("parquet.error.read", StringUtils.defaultIfBlank(
+ e.getMessage(), e.getClass().getSimpleName())));
+ startButton.setDisable(true);
+ }
+ });
+ }
+
+ private void showPreview(TransferService.ParquetPreview preview) {
+ previewTable.getColumns().clear();
+ previewTable.getItems().clear();
+ if (preview == null) {
+ previewLabel.setText(Messages.get("parquet.files.hint"));
+ return;
+ }
+ List columns = preview.getColumns();
+ for (int i = 0; i < columns.size(); i++) {
+ int index = i;
+ TableColumn, String> column = new TableColumn<>(columns.get(i)[0]);
+ Label header = new Label(columns.get(i)[0] + "\n" + columns.get(i)[1].toLowerCase(Locale.ENGLISH));
+ column.setText(null);
+ column.setGraphic(header);
+ column.setCellValueFactory(features -> new ReadOnlyObjectWrapper<>(index < features.getValue().size()
+ ? features.getValue().get(index) : null));
+ column.setPrefWidth(120);
+ previewTable.getColumns().add(column);
+ }
+ previewTable.getItems().setAll(preview.getFirstRows());
+ previewLabel.setText(Messages.format("parquet.preview", columns.size(), preview.getRows(),
+ preview.getFirstRows().size()));
+ }
+
+ private boolean isReady() {
+ return filesValid && targetCombo.getValue() != null && StringUtils.isNotBlank(tableField.getText());
+ }
+
+ private void validate() {
+ String warning = "";
+ if (context.getProfileRegistry().getProfiles().isEmpty()) {
+ warning = Messages.get("export.error.noConnections");
+ } else if (!fileList.getItems().isEmpty() && filesValid && StringUtils.isBlank(tableField.getText())) {
+ warning = Messages.get("parquet.error.table");
+ }
+ warningLabel.setText(warning);
+ if (startButton != null) {
+ startButton.setDisable(!isReady());
+ }
+ }
+
+ private void loadCatalogs(ConnectionProfile profile) {
+ catalogCombo.getItems().clear();
+ schemaCombo.getItems().clear();
+ if (profile == null) {
+ return;
+ }
+ TaskRunner.run(() -> context.getSessionManager().getSession(profile).getCatalogs(), catalogs -> {
+ catalogCombo.setDisable(catalogs.isEmpty());
+ catalogCombo.getItems().setAll(catalogs);
+ if (catalogs.isEmpty()) {
+ schemaCombo.setDisable(!profile.getDbType().isSchemaSupported());
+ if (profile.getDbType().isSchemaSupported()) {
+ loadSchemas(profile, null);
+ }
+ return;
+ }
+ catalogCombo.setValue(catalogs.contains(profile.getDatabase()) ? profile.getDatabase()
+ : catalogs.get(0));
+ schemaCombo.setDisable(!profile.getDbType().isSchemaSupported());
+ }, this::showLoadError);
+ }
+
+ /**
+ * Errors of loading after the dialog is closed are not shown
+ */
+ private void showLoadError(Throwable e) {
+ if (isShowing()) {
+ Dialogs.showError(getDialogPane().getScene().getWindow(), Messages.get("export.loadError"), e);
+ }
+ }
+
+ private void loadSchemas(ConnectionProfile profile, String catalog) {
+ schemaCombo.setDisable(false);
+ TaskRunner.run(() -> context.getSessionManager().getSession(profile).getSchemas(catalog),
+ schemas -> schemaCombo.getItems().setAll(schemas),
+ this::showLoadError);
+ }
+
+ ParquetRequest buildRequest() {
+ return new ParquetRequest(new ArrayList<>(fileList.getItems()), tableField.getText().trim(),
+ appendRadio.isSelected() ? ParquetImporter.Mode.APPEND : ParquetImporter.Mode.CREATE,
+ targetCombo.getValue(), catalogCombo.isDisabled() ? null : catalogCombo.getValue(),
+ schemaCombo.isDisabled() ? null : StringUtils.trimToNull(schemaCombo.getEditor().getText()));
+ }
+}
diff --git a/src/main/java/com/github/sqljam/face/view/LoginView.java b/src/main/java/com/github/sqljam/face/view/LoginView.java
index 9468958..346c513 100644
--- a/src/main/java/com/github/sqljam/face/view/LoginView.java
+++ b/src/main/java/com/github/sqljam/face/view/LoginView.java
@@ -16,13 +16,16 @@
package com.github.sqljam.face.view;
import java.io.IOException;
+import java.util.Comparator;
import java.util.function.Consumer;
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.face.model.ConnectionProfile;
+import com.github.sqljam.face.service.ExampleDatabase;
import javafx.collections.FXCollections;
import javafx.collections.ObservableList;
import javafx.collections.transformation.FilteredList;
+import javafx.collections.transformation.SortedList;
import javafx.geometry.Insets;
import javafx.geometry.Pos;
import javafx.scene.control.Button;
@@ -31,6 +34,7 @@
import javafx.scene.control.ScrollPane;
import javafx.scene.control.Separator;
import javafx.scene.control.TextField;
+import javafx.scene.control.Tooltip;
import javafx.scene.image.ImageView;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.HBox;
@@ -50,7 +54,10 @@ public class LoginView extends StackPane {
private final AppContext context;
private final ObservableList profiles = FXCollections.observableArrayList();
private final FilteredList filteredProfiles = new FilteredList<>(profiles, profile -> true);
- private final ListView profileList = new ListView<>(filteredProfiles);
+ // Profiles are grouped by category: relational databases, OLAP databases
+ private final SortedList sortedProfiles = new SortedList<>(filteredProfiles,
+ Comparator.comparing(DbTypeCells::categoryOf));
+ private final ListView profileList = new ListView<>(sortedProfiles);
private final ConnectionForm connectionForm = new ConnectionForm();
private final Button loginButton = new Button(Messages.get("login.login"), Icons.of(Icons.CONNECT));
private final Button testButton = new Button(Messages.get("connection.test"));
@@ -94,7 +101,16 @@ public LoginView(AppContext context, Consumer onLogin) {
deleteButton.setId("deleteButton");
deleteButton.disableProperty().bind(profileList.getSelectionModel().selectedItemProperty().isNull());
deleteButton.setOnAction(event -> deleteSelected());
- HBox listButtons = new HBox(8, newButton, deleteButton);
+ // The example database comes back after the user removed it
+ Button exampleButton = new Button(Messages.get("example.button"), Icons.of(Icons.CATALOG));
+ exampleButton.setId("exampleButton");
+ exampleButton.setTooltip(new Tooltip(Messages.get("example.button.tooltip")));
+ exampleButton.setOnAction(event -> ExampleActions.restore(getScene().getWindow(), context, profile -> {
+ reloadProfiles();
+ profileList.getItems().stream().filter(ExampleDatabase::isExample).findFirst()
+ .ifPresent(profileList.getSelectionModel()::select);
+ }));
+ HBox listButtons = new HBox(8, newButton, deleteButton, exampleButton);
Label savedLabel = new Label(Messages.get("login.savedConnections"));
savedLabel.getStyleClass().add("section-title");
VBox left = new VBox(10, savedLabel, searchField, profileList, listButtons);
@@ -109,7 +125,11 @@ public LoginView(AppContext context, Consumer onLogin) {
Label subtitle = new Label(Messages.get("login.subtitle"));
subtitle.getStyleClass().add("login-subtitle");
VBox header = new VBox(2, title, subtitle);
- HBox headerBox = new HBox(16, logo, header);
+ // The GitHub mark of the source repository at the top right, the card keeps its size
+ Region headerSpacer = new Region();
+ HBox.setHgrow(headerSpacer, Priority.ALWAYS);
+ Label repositoryMark = Branding.repositoryMark(context::openDocument);
+ HBox headerBox = new HBox(16, logo, header, headerSpacer, repositoryMark);
headerBox.setAlignment(Pos.CENTER_LEFT);
testButton.setOnAction(event -> connectionForm.testConnection(null));
diff --git a/src/main/java/com/github/sqljam/face/view/MainView.java b/src/main/java/com/github/sqljam/face/view/MainView.java
index a9a3236..3bfa556 100644
--- a/src/main/java/com/github/sqljam/face/view/MainView.java
+++ b/src/main/java/com/github/sqljam/face/view/MainView.java
@@ -15,14 +15,18 @@
*/
package com.github.sqljam.face.view;
+import java.io.File;
import java.io.IOException;
import java.util.Optional;
import com.github.sqljam.face.model.ConnectionProfile;
import com.github.sqljam.face.model.TransferRequest;
import com.github.sqljam.face.service.TransferService;
-import com.github.sqljam.impexp.ScriptImporter;
+import com.github.sqljam.impexp.AbstractFileImporter;
+import com.github.sqljam.impexp.ParquetImporter;
+import com.github.sqljam.impexp.TableQuery;
import javafx.application.Platform;
+import javafx.beans.binding.Bindings;
import javafx.geometry.Insets;
import javafx.geometry.Pos;
import javafx.scene.control.Button;
@@ -32,6 +36,7 @@
import javafx.scene.control.MenuItem;
import javafx.scene.control.ProgressIndicator;
import javafx.scene.control.RadioMenuItem;
+import javafx.scene.control.Separator;
import javafx.scene.control.SeparatorMenuItem;
import javafx.scene.control.SplitPane;
import javafx.scene.control.Tab;
@@ -41,8 +46,8 @@
import javafx.scene.control.Tooltip;
import javafx.scene.image.ImageView;
import javafx.scene.input.KeyCode;
-import javafx.scene.input.KeyCombination;
import javafx.scene.input.KeyCodeCombination;
+import javafx.scene.input.KeyCombination;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.HBox;
import javafx.scene.layout.Priority;
@@ -50,7 +55,10 @@
import javafx.scene.layout.StackPane;
import javafx.scene.layout.VBox;
import javafx.scene.text.TextAlignment;
+import javafx.stage.DirectoryChooser;
+import javafx.stage.Stage;
import javafx.stage.Window;
+import javafx.stage.WindowEvent;
/**
* @Description: MainView is the main window: connection tree on the left and opened tables on the right
@@ -72,15 +80,17 @@ public MainView(AppContext context) {
connectionTree.setOnOpenTable(this::openTable);
connectionTree.setOnExport(this::showExportDialog);
connectionTree.setOnImportScripts(this::showImportDialog);
+ connectionTree.setOnImportParquet(this::showImportParquetDialog);
connectionTree.setOnEditConnection(node -> editConnection(node.getProfile()));
connectionTree.setOnDeleteConnection(node -> deleteConnection(node.getProfile()));
connectionTree.setOnStatus(statusLabel::setText);
+ connectionTree.setOnDisconnect(this::closeTabs);
tabPane.setTabClosingPolicy(TabPane.TabClosingPolicy.ALL_TABS);
tabPane.setId("tableTabs");
statusLabel.setId("statusLabel");
StackPane center = new StackPane(createWelcome(), tabPane);
- tabPane.visibleProperty().bind(javafx.beans.binding.Bindings.isNotEmpty(tabPane.getTabs()));
+ tabPane.visibleProperty().bind(Bindings.isNotEmpty(tabPane.getTabs()));
SplitPane splitPane = new SplitPane(connectionTree, center);
splitPane.setDividerPositions(0.24);
SplitPane.setResizableWithParent(connectionTree, false);
@@ -119,18 +129,17 @@ private MenuBar createMenuBar() {
MenuItem importScripts = new MenuItem(Messages.get("action.importScripts"), Icons.of(Icons.IMPORT));
importScripts.setAccelerator(new KeyCodeCombination(KeyCode.I, KeyCombination.SHORTCUT_DOWN));
importScripts.setOnAction(event -> showImportDialog(connectionTree.getSelectedNode()));
- MenuItem exit = new MenuItem(Messages.get("action.exit"));
- exit.setOnAction(event -> {
- Window window = getWindow();
- if (window != null) {
- window.fireEvent(new javafx.stage.WindowEvent(window, javafx.stage.WindowEvent.WINDOW_CLOSE_REQUEST));
- if (window.isShowing()) {
- ((javafx.stage.Stage) window).close();
- }
- }
- });
+ MenuItem importParquet = new MenuItem(Messages.get("action.importParquet"), Icons.of(Icons.IMPORT));
+ importParquet.setId("importParquetMenuItem");
+ importParquet.setOnAction(event -> showImportParquetDialog(connectionTree.getSelectedNode()));
+ MenuItem openPackage = new MenuItem(Messages.get("package.open"), Icons.of(Icons.FOLDER));
+ openPackage.setId("openPackageMenuItem");
+ openPackage.setAccelerator(new KeyCodeCombination(KeyCode.O, KeyCombination.SHORTCUT_DOWN));
+ openPackage.setOnAction(event -> openPackage());
+ MenuItem exit = new MenuItem(Messages.get("action.exit"), Icons.of(Icons.EXIT));
+ exit.setOnAction(event -> exit());
Menu fileMenu = new Menu(Messages.get("menu.file"), null, newConnection, new SeparatorMenuItem(), export,
- importScripts, new SeparatorMenuItem(), exit);
+ importScripts, importParquet, new SeparatorMenuItem(), openPackage, new SeparatorMenuItem(), exit);
Menu themeMenu = new Menu(Messages.get("menu.theme"), Icons.of(Icons.THEME));
ToggleGroup themeGroup = new ToggleGroup();
@@ -149,9 +158,16 @@ private MenuBar createMenuBar() {
refresh.setOnAction(event -> connectionTree.refreshSelected());
Menu viewMenu = new Menu(Messages.get("menu.view"), null, themeMenu, new SeparatorMenuItem(), refresh);
+ MenuItem restoreExample = new MenuItem(Messages.get("action.restoreExample"), Icons.of(Icons.CATALOG));
+ restoreExample.setId("restoreExampleMenuItem");
+ restoreExample.setOnAction(event -> ExampleActions.restore(getWindow(), context, profile -> {
+ // Tabs of the old example database are closed
+ closeTabs(profile);
+ selectConnection(profile);
+ }));
MenuItem about = new MenuItem(Messages.get("action.about"), Icons.of(Icons.INFO));
- about.setOnAction(event -> Dialogs.showAbout(getWindow()));
- Menu helpMenu = new Menu(Messages.get("menu.help"), null, about);
+ about.setOnAction(event -> Dialogs.showAbout(getWindow(), context::openDocument));
+ Menu helpMenu = new Menu(Messages.get("menu.help"), null, restoreExample, new SeparatorMenuItem(), about);
MenuBar menuBar = new MenuBar(fileMenu, viewMenu, helpMenu);
menuBar.setUseSystemMenuBar(true);
return menuBar;
@@ -196,10 +212,28 @@ private ToolBar createToolBar() {
mark.setId("toolbarLogo");
StackPane markBox = new StackPane(mark);
markBox.setPadding(new Insets(0, 4, 0, 6));
- return new ToolBar(markBox, new javafx.scene.control.Separator(),
+ // Exit at the right end
+ Region spacer = new Region();
+ HBox.setHgrow(spacer, Priority.ALWAYS);
+ Button exitButton = toolButton(Icons.EXIT, "action.exit", this::exit);
+ exitButton.setText(Messages.get("action.exit"));
+ return new ToolBar(markBox, new Separator(),
toolButton(Icons.ADD, "action.newConnection", this::newConnection), editButton,
deleteButton, toolButton(Icons.REFRESH, "action.refresh", connectionTree::refreshSelected),
- new javafx.scene.control.Separator(), exportButton, importButton);
+ new Separator(), exportButton, importButton, spacer, exitButton);
+ }
+
+ /**
+ * Closes the window like its close button, running transfers are confirmed by the application
+ */
+ private void exit() {
+ Window window = getWindow();
+ if (window != null) {
+ window.fireEvent(new WindowEvent(window, WindowEvent.WINDOW_CLOSE_REQUEST));
+ if (window.isShowing()) {
+ ((Stage) window).close();
+ }
+ }
}
private HBox createStatusBar() {
@@ -207,11 +241,12 @@ private HBox createStatusBar() {
indicator.setPrefSize(14, 14);
indicator.visibleProperty().bind(TaskRunner.runningProperty().greaterThan(0));
Label taskLabel = new Label();
- taskLabel.textProperty().bind(javafx.beans.binding.Bindings.when(TaskRunner.runningProperty().greaterThan(0))
+ taskLabel.textProperty().bind(Bindings.when(TaskRunner.runningProperty().greaterThan(0))
.then(Messages.get("status.working")).otherwise(""));
Region spacer = new Region();
HBox.setHgrow(spacer, Priority.ALWAYS);
- HBox statusBar = new HBox(8, statusLabel, spacer, taskLabel, indicator);
+ HBox statusBar = new HBox(8, statusLabel, spacer, taskLabel, indicator,
+ Branding.repositoryMark(context::openDocument));
statusBar.setAlignment(Pos.CENTER_LEFT);
statusBar.getStyleClass().add("status-bar");
return statusBar;
@@ -249,15 +284,22 @@ private void deleteConnection(ConnectionProfile profile) {
try {
context.getProfileRegistry().removeProfile(profile.getId());
context.getSessionManager().closeSession(profile.getId());
- tabPane.getTabs().removeIf(tab -> tab.getUserData() instanceof String
- && ((String) tab.getUserData()).startsWith(profile.getId() + "|"));
+ closeTabs(profile);
connectionTree.reloadConnections();
} catch (IOException e) {
Dialogs.showError(getWindow(), Messages.get("connection.save.error"), e);
}
}
- private void openTable(DbNode node) {
+ /**
+ * Closes table tabs of the data source, e.g. after it is disconnected or deleted
+ */
+ private void closeTabs(ConnectionProfile profile) {
+ tabPane.getTabs().removeIf(tab -> tab.getUserData() instanceof String
+ && ((String) tab.getUserData()).startsWith(profile.getId() + "|"));
+ }
+
+ void openTable(DbNode node) {
String key = node.getProfile().getId() + "|" + TableTab.getKey(node);
for (Tab tab : tabPane.getTabs()) {
if (key.equals(tab.getUserData())) {
@@ -266,18 +308,28 @@ private void openTable(DbNode node) {
}
}
TableTab tab = new TableTab(context, node);
+ tab.setOnExport(this::showExportDialog);
tab.setUserData(key);
tabPane.getTabs().add(tab);
tabPane.getSelectionModel().select(tab);
}
private void showExportDialog(DbNode node) {
+ showExportDialog(node, null);
+ }
+
+ /**
+ * Export of the selected node, rows of a table may be narrowed by the query of its data pane
+ */
+ private void showExportDialog(DbNode node, TableQuery query) {
if (context.getProfileRegistry().getProfiles().isEmpty()) {
Dialogs.showInfo(getWindow(), Messages.get("export.title"), Messages.get("export.error.noConnections"));
return;
}
- ExportDialog dialog = new ExportDialog(getWindow(), context, node);
+ ExportDialog dialog = new ExportDialog(getWindow(), context, node, query);
Optional request = dialog.showAndWait();
+ // Data sources created as the target in the dialog
+ connectionTree.addNewConnections();
request.ifPresent(transferRequest -> {
int totalTables = transferRequest.getTables().isEmpty() ? dialog.getListedTableCount()
: transferRequest.getTables().size();
@@ -301,12 +353,10 @@ private void showExportDialog(DbNode node) {
}
private void showImportDialog(DbNode node) {
- if (context.getProfileRegistry().getProfiles().isEmpty()) {
- Dialogs.showInfo(getWindow(), Messages.get("import.title"), Messages.get("export.error.noConnections"));
- return;
- }
+ // Without data sources, the target is created in the dialog
Optional request = new ImportPackageDialog(getWindow(), context, node)
.showAndWait();
+ connectionTree.addNewConnections();
request.ifPresent(importRequest -> {
int totalTables = importRequest.getManifest() != null ? importRequest.getManifest().getTables().size()
: 0;
@@ -314,10 +364,42 @@ private void showImportDialog(DbNode node) {
"progress.importTitle", importRequest.getTarget().getName()), totalTables);
progress.run(() -> context.getTransferService().importScripts(importRequest.getTarget(),
importRequest.getCatalog(), importRequest.getSchema(), importRequest.getDirectory(),
- importRequest.isStopOnError(), progress.getListener()), (ScriptImporter importer) ->
+ importRequest.isStopOnError(), progress.getListener()), (AbstractFileImporter importer) ->
Messages.format("import.summary", importer.getExecutedCount(), importer.getLobCount(),
importer.getFailedCount()), () -> context.getSessionManager()
.closeSession(importRequest.getTarget().getId()));
});
}
+
+ private void showImportParquetDialog(DbNode node) {
+ Optional request = new ImportParquetDialog(getWindow(), context, node)
+ .showAndWait();
+ connectionTree.addNewConnections();
+ request.ifPresent(parquetRequest -> {
+ ProgressDialog progress = new ProgressDialog(getWindow(), context, Messages.format(
+ "progress.importTitle", parquetRequest.getTarget().getName()), 1);
+ progress.run(() -> context.getTransferService().importParquet(parquetRequest.getTarget(),
+ parquetRequest.getCatalog(), parquetRequest.getSchema(), parquetRequest.getFiles(),
+ parquetRequest.getTableName(), parquetRequest.getMode(), progress.getListener()),
+ (ParquetImporter importer) -> Messages.format("parquet.summary", importer.getImportedRows(),
+ parquetRequest.getTableName()), () -> context.getSessionManager()
+ .closeSession(parquetRequest.getTarget().getId()));
+ });
+ }
+
+ /**
+ * Lists and previews files of an export package (sql scripts or Parquet files)
+ */
+ private void openPackage() {
+ DirectoryChooser chooser = new DirectoryChooser();
+ chooser.setTitle(Messages.get("package.chooseDirectory"));
+ String lastDirectory = context.getSettings().getLastImportDirectory();
+ if (lastDirectory != null && new File(lastDirectory).isDirectory()) {
+ chooser.setInitialDirectory(new File(lastDirectory));
+ }
+ File directory = chooser.showDialog(getWindow());
+ if (directory != null) {
+ new PackageViewer(getWindow(), context, directory).show();
+ }
+ }
}
diff --git a/src/main/java/com/github/sqljam/face/view/PackageViewer.java b/src/main/java/com/github/sqljam/face/view/PackageViewer.java
new file mode 100644
index 0000000..072f137
--- /dev/null
+++ b/src/main/java/com/github/sqljam/face/view/PackageViewer.java
@@ -0,0 +1,362 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.face.view;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Instant;
+import java.time.LocalDateTime;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.github.sqljam.face.service.TransferService;
+import com.github.sqljam.impexp.ExportManifest;
+import com.github.sqljam.impexp.ParquetExporter;
+import javafx.beans.property.ReadOnlyObjectWrapper;
+import javafx.beans.property.SimpleStringProperty;
+import javafx.geometry.Insets;
+import javafx.scene.Scene;
+import javafx.scene.control.Label;
+import javafx.scene.control.SplitPane;
+import javafx.scene.control.TableColumn;
+import javafx.scene.control.TableView;
+import javafx.scene.control.TextArea;
+import javafx.scene.control.TreeItem;
+import javafx.scene.control.TreeTableCell;
+import javafx.scene.control.TreeTableColumn;
+import javafx.scene.control.TreeTableView;
+import javafx.scene.layout.BorderPane;
+import javafx.scene.layout.Priority;
+import javafx.scene.layout.StackPane;
+import javafx.scene.layout.VBox;
+import javafx.stage.Stage;
+import javafx.stage.Window;
+
+/**
+ * @Description: PackageViewer shows the files of an export package (sql scripts or Parquet files) with sizes and
+ * modified times, and previews the selected file: text of sql and json files (manifest.json formatted), columns and first rows of
+ * Parquet files, text or hex of LOB files. Large files are previewed by their first part.
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+public class PackageViewer {
+
+ /**
+ * Bytes of a text preview
+ */
+ static final int TEXT_PREVIEW_BYTES = 256 * 1024;
+ /**
+ * Bytes of a hex preview of binary files
+ */
+ static final int HEX_PREVIEW_BYTES = 4 * 1024;
+ /**
+ * Rows of a Parquet preview
+ */
+ static final int PARQUET_PREVIEW_ROWS = 100;
+
+ private final AppContext context;
+ private final File directory;
+ private final Stage stage = new Stage();
+ private static final DateTimeFormatter MODIFIED_FORMATTER =
+ DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");
+
+ private final TreeTableView fileTree = new TreeTableView<>();
+ private final Label summaryLabel = new Label();
+ private final Label fileLabel = new Label();
+ private final TextArea textPreview = new TextArea();
+ private final TableView> tablePreview = new TableView<>();
+ private final StackPane previewPane = new StackPane(textPreview, tablePreview);
+
+ public PackageViewer(Window owner, AppContext context, File directory) {
+ this.context = context;
+ this.directory = directory;
+ stage.initOwner(owner);
+ Branding.applyIcons(stage);
+ stage.setTitle(Messages.format("package.title", directory.getName()));
+
+ fileTree.setId("packageFileTree");
+ summaryLabel.setId("packageSummary");
+ fileLabel.setId("packageFileLabel");
+ textPreview.setId("packageTextPreview");
+ tablePreview.setId("packageTablePreview");
+ summaryLabel.getStyleClass().add("muted");
+ summaryLabel.setWrapText(true);
+ fileLabel.getStyleClass().add("section-title");
+ textPreview.setEditable(false);
+ textPreview.getStyleClass().add("mono");
+ tablePreview.setPlaceholder(new Label(Messages.get("package.noRows")));
+ showText("");
+
+ fileTree.setRoot(createItem(directory));
+ fileTree.getRoot().setExpanded(true);
+ setupColumns();
+ fileTree.getSelectionModel().selectedItemProperty().addListener((obs, oldItem, item) -> {
+ if (item != null && item.getValue().isFile()) {
+ preview(item.getValue());
+ }
+ });
+ summaryLabel.setText(describe(directory));
+
+ VBox left = new VBox(8, summaryLabel, fileTree);
+ VBox.setVgrow(fileTree, Priority.ALWAYS);
+ left.setPadding(new Insets(8));
+ VBox right = new VBox(8, fileLabel, previewPane);
+ VBox.setVgrow(previewPane, Priority.ALWAYS);
+ right.setPadding(new Insets(8));
+ SplitPane splitPane = new SplitPane(left, right);
+ splitPane.setDividerPositions(0.32);
+ BorderPane root = new BorderPane(splitPane);
+ root.setId("packageViewer");
+ Scene scene = new Scene(root, 1100, 680);
+ scene.getStylesheets().add(PackageViewer.class.getResource("/com/github/sqljam/face/app.css")
+ .toExternalForm());
+ stage.setScene(scene);
+ }
+
+ /**
+ * Name (with icon), size and modified time of files
+ */
+ private void setupColumns() {
+ TreeTableColumn nameColumn = new TreeTableColumn<>(Messages.get("package.column.name"));
+ nameColumn.setCellValueFactory(cell -> new ReadOnlyObjectWrapper<>(cell.getValue().getValue()));
+ nameColumn.setCellFactory(column -> new TreeTableCell<>() {
+ @Override
+ protected void updateItem(File item, boolean empty) {
+ super.updateItem(item, empty);
+ if (empty || item == null) {
+ setText(null);
+ setGraphic(null);
+ } else {
+ setText(item.getName());
+ setGraphic(Icons.of(item.isDirectory() ? Icons.FOLDER
+ : isParquet(item) ? Icons.TABLE : "fth-file-text"));
+ }
+ }
+ });
+ nameColumn.setPrefWidth(200);
+ TreeTableColumn sizeColumn = new TreeTableColumn<>(Messages.get("package.column.size"));
+ sizeColumn.setCellValueFactory(cell -> new ReadOnlyObjectWrapper<>(formatSize(cell.getValue().getValue())));
+ sizeColumn.setPrefWidth(80);
+ sizeColumn.setStyle("-fx-alignment: CENTER-RIGHT;");
+ TreeTableColumn modifiedColumn = new TreeTableColumn<>(
+ Messages.get("package.column.modified"));
+ modifiedColumn.setCellValueFactory(cell -> new ReadOnlyObjectWrapper<>(
+ formatModified(cell.getValue().getValue())));
+ modifiedColumn.setPrefWidth(140);
+ fileTree.getColumns().setAll(List.of(nameColumn, sizeColumn, modifiedColumn));
+ fileTree.setColumnResizePolicy(TreeTableView.CONSTRAINED_RESIZE_POLICY);
+ }
+
+ /**
+ * Size of a file, or of all files of a directory
+ */
+ static String formatSize(File file) {
+ long size = file.isDirectory() ? FileUtils.sizeOfDirectory(file) : file.length();
+ return FileUtils.byteCountToDisplaySize(size);
+ }
+
+ static String formatModified(File file) {
+ return LocalDateTime.ofInstant(Instant.ofEpochMilli(file.lastModified()),
+ ZoneId.systemDefault()).format(MODIFIED_FORMATTER);
+ }
+
+ public void show() {
+ stage.show();
+ // manifest.json is previewed first
+ selectFile(new File(directory, ExportManifest.FILE_NAME));
+ }
+
+ public Stage getStage() {
+ return stage;
+ }
+
+ /**
+ * Selects the file in the tree and previews it
+ */
+ public boolean selectFile(File file) {
+ TreeItem item = findItem(fileTree.getRoot(), file);
+ if (item != null) {
+ for (TreeItem parent = item.getParent(); parent != null; parent = parent.getParent()) {
+ parent.setExpanded(true);
+ }
+ fileTree.getSelectionModel().select(item);
+ return true;
+ }
+ return false;
+ }
+
+ private static TreeItem findItem(TreeItem item, File file) {
+ if (item.getValue().equals(file)) {
+ return item;
+ }
+ for (TreeItem child : item.getChildren()) {
+ TreeItem found = findItem(child, file);
+ if (found != null) {
+ return found;
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Files of the package, directories first, temporary workspaces are skipped
+ */
+ static TreeItem createItem(File file) {
+ TreeItem item = new TreeItem<>(file);
+ if (file.isDirectory()) {
+ File[] children = file.listFiles(child -> !child.getName().startsWith(".sqljam-workspace-")
+ && !child.isHidden());
+ if (children != null) {
+ Arrays.stream(children).sorted(Comparator.comparing((File child) -> !child.isDirectory())
+ .thenComparing(File::getName)).forEach(child -> item.getChildren().add(createItem(child)));
+ }
+ }
+ return item;
+ }
+
+ /**
+ * Summary of the package by its manifest: data format, source, target, tables and rows
+ */
+ static String describe(File directory) {
+ try {
+ ExportManifest manifest = ExportManifest.read(directory);
+ if (manifest == null) {
+ return Messages.get("package.noManifest");
+ }
+ long rows = manifest.getTables().stream().mapToLong(ExportManifest.TableEntry::getRows).sum();
+ return Messages.format("package.summary", manifest.getDataFormat(), manifest.getStatus(),
+ manifest.getSource().getDbType(), manifest.getTarget().getDbType(), manifest.getTables().size(),
+ rows);
+ } catch (IOException e) {
+ return e.getMessage();
+ }
+ }
+
+ private void preview(File file) {
+ fileLabel.setText(String.format("%s (%s, %s)", file.getName(), formatSize(file), formatModified(file)));
+ if (isParquet(file)) {
+ showText(Messages.get("package.loading"));
+ TaskRunner.run(() -> context.getTransferService().previewParquet(List.of(file), PARQUET_PREVIEW_ROWS),
+ this::showParquet, e -> showText(e.getMessage()));
+ return;
+ }
+ try {
+ showText(readPreview(file));
+ } catch (IOException e) {
+ showText(e.getMessage());
+ }
+ }
+
+ private void showText(String text) {
+ textPreview.setText(text);
+ textPreview.setVisible(true);
+ tablePreview.setVisible(false);
+ }
+
+ private void showParquet(TransferService.ParquetPreview preview) {
+ tablePreview.getColumns().clear();
+ List columns = preview.getColumns();
+ for (int i = 0; i < columns.size(); i++) {
+ int index = i;
+ TableColumn, String> column = new TableColumn<>(columns.get(i)[0] + "\n" + columns.get(i)[1]);
+ column.setCellValueFactory(cell -> new SimpleStringProperty(index < cell.getValue().size()
+ ? cell.getValue().get(index) : null));
+ column.setPrefWidth(140);
+ tablePreview.getColumns().add(column);
+ }
+ tablePreview.getItems().setAll(preview.getFirstRows());
+ fileLabel.setText(fileLabel.getText() + " " + Messages.format("package.parquetRows", preview.getRows(),
+ preview.getFirstRows().size()));
+ textPreview.setVisible(false);
+ tablePreview.setVisible(true);
+ }
+
+ static boolean isParquet(File file) {
+ return file.getName().toLowerCase(Locale.ENGLISH).endsWith(ParquetExporter.PARQUET_EXTENSION);
+ }
+
+ /**
+ * Preview of a non Parquet file: manifest json formatted, text files, hex of binary files
+ */
+ static String readPreview(File file) throws IOException {
+ String name = file.getName().toLowerCase(Locale.ENGLISH);
+ if (name.endsWith(".json") && file.length() <= TEXT_PREVIEW_BYTES) {
+ ObjectMapper mapper = new ObjectMapper().enable(SerializationFeature.INDENT_OUTPUT);
+ try {
+ return mapper.writeValueAsString(mapper.readTree(file));
+ } catch (IOException e) {
+ return readText(file, TEXT_PREVIEW_BYTES);
+ }
+ }
+ if (name.endsWith(".blob")) {
+ return hexPreview(file, HEX_PREVIEW_BYTES);
+ }
+ return readText(file, TEXT_PREVIEW_BYTES);
+ }
+
+ /**
+ * Text of the first bytes of a file, a notice is appended if the file is larger
+ */
+ static String readText(File file, int maxBytes) throws IOException {
+ byte[] bytes;
+ try (InputStream in = Files.newInputStream(file.toPath())) {
+ bytes = in.readNBytes(maxBytes);
+ }
+ String text = new String(bytes, StandardCharsets.UTF_8);
+ if (file.length() > maxBytes) {
+ text += "\n\n" + Messages.format("package.truncated", FileUtils.byteCountToDisplaySize(maxBytes),
+ FileUtils.byteCountToDisplaySize(file.length()));
+ }
+ return text;
+ }
+
+ /**
+ * Hex dump of the first bytes of a binary file, 16 bytes per line
+ */
+ static String hexPreview(File file, int maxBytes) throws IOException {
+ byte[] bytes;
+ try (InputStream in = Files.newInputStream(file.toPath())) {
+ bytes = in.readNBytes(maxBytes);
+ }
+ StringBuilder hex = new StringBuilder();
+ for (int offset = 0; offset < bytes.length; offset += 16) {
+ hex.append(String.format("%08x ", offset));
+ List parts = new ArrayList<>();
+ for (int i = offset; i < Math.min(offset + 16, bytes.length); i++) {
+ parts.add(String.format("%02x", bytes[i]));
+ }
+ hex.append(StringUtils.join(parts, ' ')).append('\n');
+ }
+ if (file.length() > maxBytes) {
+ hex.append('\n').append(Messages.format("package.truncated", FileUtils.byteCountToDisplaySize(maxBytes),
+ FileUtils.byteCountToDisplaySize(file.length())));
+ }
+ return hex.toString();
+ }
+}
diff --git a/src/main/java/com/github/sqljam/face/view/ProgressDialog.java b/src/main/java/com/github/sqljam/face/view/ProgressDialog.java
index 8178714..495b755 100644
--- a/src/main/java/com/github/sqljam/face/view/ProgressDialog.java
+++ b/src/main/java/com/github/sqljam/face/view/ProgressDialog.java
@@ -22,7 +22,6 @@
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.impexp.ExportCancelledException;
-import atlantafx.base.theme.Styles;
import com.github.sqljam.impexp.ExportListener;
import javafx.animation.KeyFrame;
import javafx.animation.Timeline;
@@ -46,6 +45,7 @@
import javafx.stage.Window;
import javafx.util.Duration;
import lombok.extern.slf4j.Slf4j;
+import atlantafx.base.theme.Styles;
/**
* @Description: ProgressDialog shows the progress of an export or import, it can be cancelled
@@ -70,6 +70,7 @@ public class ProgressDialog {
private final ListView logList = new ListView<>();
private final Button cancelButton = new Button(Messages.get("button.cancel"), Icons.of(Icons.CANCEL));
private final Button openFolderButton = new Button(Messages.get("progress.openFolder"), Icons.of(Icons.OPEN));
+ private final Button viewPackageButton = new Button(Messages.get("package.view"), Icons.of(Icons.TABLE));
private final Label manifestLabel = new Label();
private final AtomicBoolean cancelled = new AtomicBoolean();
private final int totalTables;
@@ -108,6 +109,7 @@ public ProgressDialog(Window owner, AppContext context, String title, int totalT
logList.setId("logList");
cancelButton.setId("cancelButton");
openFolderButton.setId("openFolderButton");
+ viewPackageButton.setId("viewPackageButton");
manifestLabel.setId("manifestLabel");
statusLabel.getStyleClass().add("progress-status");
@@ -151,6 +153,13 @@ protected void updateItem(LogEntry item, boolean empty) {
});
openFolderButton.setVisible(false);
openFolderButton.setManaged(false);
+ viewPackageButton.setVisible(false);
+ viewPackageButton.setManaged(false);
+ viewPackageButton.setOnAction(event -> {
+ if (outputDirectory != null) {
+ new PackageViewer(stage, context, outputDirectory).show();
+ }
+ });
openFolderButton.setOnAction(event -> {
if (outputDirectory != null) {
context.getHostServices().showDocument(outputDirectory.toURI().toString());
@@ -162,10 +171,11 @@ protected void updateItem(LogEntry item, boolean empty) {
manifestLabel.setVisible(false);
manifestLabel.setManaged(false);
openFolderButton.setMinWidth(Region.USE_PREF_SIZE);
+ viewPackageButton.setMinWidth(Region.USE_PREF_SIZE);
cancelButton.setMinWidth(Region.USE_PREF_SIZE);
Region spacer = new Region();
HBox.setHgrow(spacer, Priority.ALWAYS);
- HBox buttons = new HBox(8, spacer, openFolderButton, cancelButton);
+ HBox buttons = new HBox(8, spacer, viewPackageButton, openFolderButton, cancelButton);
buttons.setAlignment(Pos.CENTER_RIGHT);
Region headerSpacer = new Region();
@@ -387,6 +397,8 @@ private void finish(String status, boolean failed, boolean cancelled, String sum
if (!failed && outputDirectory != null) {
openFolderButton.setVisible(true);
openFolderButton.setManaged(true);
+ viewPackageButton.setVisible(true);
+ viewPackageButton.setManaged(true);
File manifest = new File(outputDirectory, "manifest.json");
if (manifest.exists()) {
manifestLabel.setText(Messages.format("progress.manifest", manifest.getAbsolutePath()));
diff --git a/src/main/java/com/github/sqljam/face/view/SessionManager.java b/src/main/java/com/github/sqljam/face/view/SessionManager.java
index fd5ac4a..82426e5 100644
--- a/src/main/java/com/github/sqljam/face/view/SessionManager.java
+++ b/src/main/java/com/github/sqljam/face/view/SessionManager.java
@@ -45,6 +45,16 @@ public void closeSession(String profileId) {
}
}
+ /**
+ * Closes the session in the calling thread, e.g. before the database file of the profile is replaced
+ */
+ public void closeSessionNow(String profileId) {
+ DatabaseSession session = sessions.remove(profileId);
+ if (session != null) {
+ session.close();
+ }
+ }
+
public void closeAll() {
sessions.values().forEach(DatabaseSession::close);
sessions.clear();
diff --git a/src/main/java/com/github/sqljam/face/view/TableTab.java b/src/main/java/com/github/sqljam/face/view/TableTab.java
index 52151ca..9f240b8 100644
--- a/src/main/java/com/github/sqljam/face/view/TableTab.java
+++ b/src/main/java/com/github/sqljam/face/view/TableTab.java
@@ -16,8 +16,10 @@
package com.github.sqljam.face.view;
import java.util.List;
+import java.util.function.BiConsumer;
import java.util.function.Function;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.face.model.ColumnInfo;
@@ -25,6 +27,7 @@
import com.github.sqljam.face.model.IndexInfo;
import com.github.sqljam.face.service.DatabaseSession;
import com.github.sqljam.impexp.DbType;
+import com.github.sqljam.impexp.TableQuery;
import javafx.beans.property.ReadOnlyObjectWrapper;
import javafx.beans.value.ObservableValue;
import javafx.geometry.Insets;
@@ -37,6 +40,7 @@
import javafx.scene.control.TableColumn;
import javafx.scene.control.TableView;
import javafx.scene.control.TextArea;
+import javafx.scene.control.Tooltip;
import javafx.scene.input.Clipboard;
import javafx.scene.input.ClipboardContent;
import javafx.scene.layout.BorderPane;
@@ -65,12 +69,23 @@ public class TableTab extends Tab {
private boolean ddlLoaded;
private boolean dataLoaded;
+ /**
+ * Rows narrowed in the data pane are exported by the consumer
+ */
+ public void setOnExport(BiConsumer onExport) {
+ dataPane.setOnExport(query -> onExport.accept(node, query));
+ }
+
+ DataPane getDataPane() {
+ return dataPane;
+ }
+
public TableTab(AppContext context, DbNode node) {
this.node = node;
this.session = context.getSessionManager().getSession(node.getProfile());
setText(node.getTable().getName());
setGraphic(Icons.of(Icons.TABLE));
- setTooltip(new javafx.scene.control.Tooltip(getKey(node)));
+ setTooltip(new Tooltip(getKey(node)));
dataPane = new DataPane(session, node.getCatalog(), node.getSchema(), node.getTable().getName(),
context.getSettings().getDataPageSize());
@@ -102,7 +117,7 @@ public TableTab(AppContext context, DbNode node) {
* Unique key of the table, an opened table is reused
*/
public static String getKey(DbNode node) {
- return java.util.stream.Stream.of(node.getProfile().getName(), node.getCatalog(), node.getSchema(),
+ return Stream.of(node.getProfile().getName(), node.getCatalog(), node.getSchema(),
node.getTable().getName()).filter(StringUtils::isNotBlank).collect(Collectors.joining("."));
}
diff --git a/src/main/java/com/github/sqljam/face/view/ThemeManager.java b/src/main/java/com/github/sqljam/face/view/ThemeManager.java
index a5b405f..2726b4d 100644
--- a/src/main/java/com/github/sqljam/face/view/ThemeManager.java
+++ b/src/main/java/com/github/sqljam/face/view/ThemeManager.java
@@ -21,6 +21,7 @@
import java.util.Map;
import java.util.function.Supplier;
+import javafx.application.Application;
import atlantafx.base.theme.CupertinoDark;
import atlantafx.base.theme.CupertinoLight;
import atlantafx.base.theme.Dracula;
@@ -29,7 +30,6 @@
import atlantafx.base.theme.PrimerDark;
import atlantafx.base.theme.PrimerLight;
import atlantafx.base.theme.Theme;
-import javafx.application.Application;
/**
* @Description: ThemeManager switches AtlantaFX themes of the application
diff --git a/src/main/java/com/github/sqljam/impexp/AbstractFileImporter.java b/src/main/java/com/github/sqljam/impexp/AbstractFileImporter.java
new file mode 100644
index 0000000..69b7434
--- /dev/null
+++ b/src/main/java/com/github/sqljam/impexp/AbstractFileImporter.java
@@ -0,0 +1,258 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.impexp;
+
+import java.io.File;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.SQLException;
+
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * @Description: AbstractFileImporter imports files of an export package into a database by a connection:
+ * manifest.json is validated (format, status, target database type and checksums of files), files are
+ * imported in its order with the overall progress in bytes. {@link ScriptImporter} imports packages
+ * of sql scripts, {@link ParquetImporter} packages of Parquet files and Parquet files of other tools.
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+public abstract class AbstractFileImporter {
+
+ protected final Connection connection;
+ protected final DbType dbType;
+ protected boolean stopOnError = true;
+ protected int batchSize = SqlScriptRunner.DEFAULT_BATCH_SIZE;
+ protected ExportListener exportListener = ExportListener.NONE;
+ protected File workspaceDirectory;
+ protected int executedCount;
+ protected int failedCount;
+ /**
+ * Overall progress in bytes of imported files
+ */
+ protected long totalBytes;
+ protected long completedBytes;
+
+ protected AbstractFileImporter(Connection connection, DbType dbType) {
+ this.connection = connection;
+ this.dbType = dbType;
+ }
+
+ /**
+ * Importer of an export directory by the data format of its manifest: Parquet packages are imported by
+ * {@link ParquetImporter}, sql scripts (with or without manifest) by {@link ScriptImporter}
+ */
+ public static AbstractFileImporter forDirectory(File dir, Connection connection, DbType dbType)
+ throws IOException {
+ ExportManifest manifest = ExportManifest.read(dir);
+ if (manifest != null && manifest.getDataFormat() == DataFormat.PARQUET) {
+ return new ParquetImporter(connection, dbType);
+ }
+ return new ScriptImporter(connection, dbType);
+ }
+
+ public void setStopOnError(boolean stopOnError) {
+ this.stopOnError = stopOnError;
+ }
+
+ /**
+ * INSERT statements per batch of sql scripts, 1 or less executes statement by statement
+ */
+ public void setBatchSize(int batchSize) {
+ this.batchSize = batchSize;
+ }
+
+ public void setExportListener(ExportListener exportListener) {
+ this.exportListener = exportListener != null ? exportListener : ExportListener.NONE;
+ }
+
+ /**
+ * Directory of temporary files (the DuckDB workspace of Parquet files), the system temporary directory by default
+ */
+ public void setWorkspaceDirectory(File workspaceDirectory) {
+ this.workspaceDirectory = workspaceDirectory;
+ }
+
+ /**
+ * Executed sql statements and imported rows of Parquet files
+ */
+ public int getExecutedCount() {
+ return executedCount;
+ }
+
+ public int getFailedCount() {
+ return failedCount;
+ }
+
+ /**
+ * Restored LOB values, LOBs of Parquet packages are inside Parquet files
+ */
+ public int getLobCount() {
+ return 0;
+ }
+
+ /**
+ * Imports an export directory. Files are imported in the order of manifest.json if it exists.
+ */
+ public void importDirectory(File dir) throws IOException, SQLException {
+ ExportManifest manifest = ExportManifest.read(dir);
+ if (manifest == null) {
+ importDirectoryWithoutManifest(dir);
+ return;
+ }
+ boolean successful = false;
+ try {
+ validateManifest(dir, manifest);
+ startProgress(getManifestBytes(dir, manifest));
+ importManifest(dir, manifest);
+ successful = true;
+ } finally {
+ exportListener.onEnd(successful);
+ }
+ }
+
+ /**
+ * Imports files listed by the manifest in order, the manifest is validated already
+ */
+ protected abstract void importManifest(File dir, ExportManifest manifest) throws IOException, SQLException;
+
+ /**
+ * Imports a directory without manifest.json, discovered by the directory layout
+ */
+ protected void importDirectoryWithoutManifest(File dir) throws IOException, SQLException {
+ throw new ImpExpException("Manifest not found: " + new File(dir, ExportManifest.FILE_NAME));
+ }
+
+ /**
+ * Format, status, target database type and files of the manifest
+ */
+ protected void validateManifest(File dir, ExportManifest manifest) throws IOException {
+ if (!ExportManifest.FORMAT.equals(manifest.getFormat())) {
+ throw new ImpExpException("Unknown export format: " + manifest.getFormat());
+ }
+ if (manifest.getStatus() != ExportManifest.Status.COMPLETED) {
+ throw new ImpExpException("The export is not completed: " + manifest.getStatus());
+ }
+ DbType targetDbType = manifest.getTarget().getDbType();
+ if (dbType != null && targetDbType != null && dbType != targetDbType) {
+ throw new ImpExpException(String.format("The scripts are generated for %s, but the target database is %s",
+ targetDbType.getDisplayName(), dbType.getDisplayName()));
+ }
+ for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) {
+ File file = new File(dir, fileEntry.getPath());
+ if (!file.exists()) {
+ throw new ImpExpException("File not found: " + fileEntry.getPath());
+ }
+ if (StringUtils.isNotBlank(fileEntry.getSha256()) && !fileEntry.getSha256().equals(
+ ExportManifest.sha256(file))) {
+ throw new ImpExpException("File is modified or corrupted (checksum mismatch): " + fileEntry.getPath());
+ }
+ }
+ }
+
+ /**
+ * Bytes of the files of the manifest, the total of the progress
+ */
+ protected long getManifestBytes(File dir, ExportManifest manifest) {
+ long bytes = 0;
+ for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) {
+ bytes += new File(dir, fileEntry.getPath()).length();
+ }
+ return bytes;
+ }
+
+ protected void startProgress(long totalBytes) {
+ this.totalBytes = Math.max(totalBytes, 1);
+ this.completedBytes = 0;
+ exportListener.onProgress(0, this.totalBytes);
+ }
+
+ protected void completeBytes(long bytes) {
+ completedBytes += bytes;
+ exportListener.onProgress(Math.min(completedBytes, totalBytes), totalBytes);
+ }
+
+ protected static long sizeOf(File file) {
+ if (file.isDirectory()) {
+ File[] files = file.listFiles();
+ long size = 0;
+ if (files != null) {
+ for (File child : files) {
+ size += sizeOf(child);
+ }
+ }
+ return size;
+ }
+ return file.exists() ? file.length() : 0;
+ }
+
+ /**
+ * Executes a sql script (schema.sql, data files, constraints.sql), progress of statements is converted to bytes
+ */
+ protected void runScript(File file) throws IOException, SQLException {
+ if (!file.exists()) {
+ return;
+ }
+ exportListener.onMessage("Executing script: " + file.getName());
+ SqlScriptRunner runner = new SqlScriptRunner(connection);
+ runner.setStopOnError(stopOnError);
+ runner.setBatchSize(batchSize);
+ long fileBytes = file.length();
+ runner.setExportListener(new ForwardingListener(exportListener) {
+
+ @Override
+ public void onProgress(long processed, long total) {
+ long bytes = total > 0 ? fileBytes * processed / total : 0;
+ exportListener.onProgress(Math.min(completedBytes + bytes, totalBytes), totalBytes);
+ }
+ });
+ try {
+ runner.runScript(file);
+ } finally {
+ executedCount += runner.getExecutedCount();
+ failedCount += runner.getFailedCount();
+ completeBytes(fileBytes);
+ }
+ }
+
+ /**
+ * Forwards messages and errors of a part of the import, the end of each part is not the end of the import
+ */
+ protected static class ForwardingListener implements ExportListener {
+
+ private final ExportListener delegate;
+
+ protected ForwardingListener(ExportListener delegate) {
+ this.delegate = delegate;
+ }
+
+ @Override
+ public void onMessage(String message) {
+ delegate.onMessage(message);
+ }
+
+ @Override
+ public void onError(String message, Throwable e) {
+ delegate.onError(message, e);
+ }
+
+ @Override
+ public boolean isCancelled() {
+ return delegate.isCancelled();
+ }
+ }
+}
diff --git a/src/main/java/com/github/sqljam/impexp/CatalogMetaData.java b/src/main/java/com/github/sqljam/impexp/CatalogMetaData.java
index 78ed266..3f3ecc0 100644
--- a/src/main/java/com/github/sqljam/impexp/CatalogMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/CatalogMetaData.java
@@ -23,9 +23,10 @@
import java.util.List;
import java.util.Map;
import java.util.Optional;
-import lombok.extern.slf4j.Slf4j;
+
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
+import lombok.extern.slf4j.Slf4j;
/**
* @Description: CatalogMetaData is a catalog (database) of the server, it loads the included schemas
diff --git a/src/main/java/com/github/sqljam/impexp/ColumnMetaData.java b/src/main/java/com/github/sqljam/impexp/ColumnMetaData.java
index 4f6cee7..455b7f4 100644
--- a/src/main/java/com/github/sqljam/impexp/ColumnMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/ColumnMetaData.java
@@ -20,6 +20,8 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import org.apache.commons.lang3.StringUtils;
@@ -62,9 +64,12 @@ public String[] getStatements() throws SQLException {
Dialect dialect = getDialect();
String catalogName = getCatalogName();
String schemaName = getSchemaName();
- String tableName = getTableName();
+ String tableName = dialect.getTargetTableName(getTableName());
int dataType = TableMetaData.getInt(detail, "DATA_TYPE");
- String typeName = (String) detail.get("TYPE_NAME");
+ String typeName = getUserTypeName(dialect);
+ if (typeName == null) {
+ typeName = (String) detail.get("TYPE_NAME");
+ }
int columnSize = TableMetaData.getInt(detail, "COLUMN_SIZE");
int columnScale = TableMetaData.getInt(detail, "DECIMAL_DIGITS");
String comment = (String) detail.get("REMARKS");
@@ -86,7 +91,7 @@ public String[] getStatements() throws SQLException {
columnSize,
columnScale, defaultValue, nullable, comment);
}
- dialect.registerColumnTypeName(tableName, columnName,
+ dialect.registerColumnTypeName(getTableName(), columnName,
dialect.getColumnTypeName(dataType, typeName, columnSize, columnScale));
return new String[]{statement};
}
@@ -119,6 +124,32 @@ public String getColumnName() {
return columnName;
}
+ /**
+ * User defined type (enum, domain) of the column for the same database type, null otherwise
+ */
+ @SuppressWarnings("unchecked")
+ private String getUserTypeName(Dialect dialect) {
+ Map userType = (Map) detail.get("USER_TYPE");
+ if (userType == null || dialect.isCrossDatabase()) {
+ return null;
+ }
+ return Dialect.VERBATIM_TYPE_PREFIX + dialect.getUserTypeName(getCatalogName(), getSchemaName(),
+ (String) userType.get("NAME"));
+ }
+
+ /**
+ * Statement creating the user defined type of the column for the same database type, null otherwise
+ */
+ @SuppressWarnings("unchecked")
+ public String getUserTypeStatement() {
+ Map userType = (Map) detail.get("USER_TYPE");
+ Dialect dialect = getDialect();
+ if (userType == null || dialect.isCrossDatabase()) {
+ return null;
+ }
+ return dialect.getCreateUserTypeStatement(getCatalogName(), getSchemaName(), userType);
+ }
+
/**
* Objects referenced by default values (e.g. sequences) are moved to the target schema
*/
@@ -130,8 +161,8 @@ private String replaceSchema(String defaultValue, Dialect dialect) {
|| dialect.isCrossDatabase()) {
return defaultValue;
}
- String quotedSchema = java.util.regex.Pattern.quote(schemaName);
- String replacement = java.util.regex.Matcher.quoteReplacement(dialect.quoteIdentifier(targetSchemaName));
+ String quotedSchema = Pattern.quote(schemaName);
+ String replacement = Matcher.quoteReplacement(dialect.quoteIdentifier(targetSchemaName));
return defaultValue.replaceAll("(?i)(\\[" + quotedSchema + "\\]|\"" + quotedSchema + "\"|`" + quotedSchema
+ "`|\\b" + quotedSchema + "\\b)(?=\\s*\\.)", replacement);
}
diff --git a/src/main/java/com/github/sqljam/impexp/CombinedIndexMetaData.java b/src/main/java/com/github/sqljam/impexp/CombinedIndexMetaData.java
index 8b4a44d..d685308 100644
--- a/src/main/java/com/github/sqljam/impexp/CombinedIndexMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/CombinedIndexMetaData.java
@@ -105,7 +105,7 @@ public String[] getStatements() throws SQLException {
}
String catalogName = getCatalogName();
String schemaName = getSchemaName();
- String tableName = getTableName();
+ String tableName = getDialect().getTargetTableName(getTableName());
boolean partition = tableMetaData.isPartitioned();
boolean unique = IndexMetaData.isUnique(getDetail());
String indexName = getDialect().getIndexNameStatement(catalogName, schemaName, tableName, columnNames, unique,
diff --git a/src/main/java/com/github/sqljam/impexp/CommentMetaData.java b/src/main/java/com/github/sqljam/impexp/CommentMetaData.java
index 926585a..d8be9af 100644
--- a/src/main/java/com/github/sqljam/impexp/CommentMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/CommentMetaData.java
@@ -46,7 +46,7 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
public String[] getStatements() throws SQLException {
String catalogName = getCatalogName();
String schemaName = getSchemaName();
- String tableName = getTableName();
+ String tableName = getDialect().getTargetTableName(getTableName());
String comment = (String) detail.get("REMARKS");
String statement;
if (columnName == null) {
diff --git a/src/main/java/com/github/sqljam/impexp/DataFormat.java b/src/main/java/com/github/sqljam/impexp/DataFormat.java
new file mode 100644
index 0000000..22bcf3f
--- /dev/null
+++ b/src/main/java/com/github/sqljam/impexp/DataFormat.java
@@ -0,0 +1,30 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.impexp;
+
+/**
+ * @Description: DataFormat of rows in an export package: SQL insert statements, or Parquet files (one file per
+ * table) for analytical tools
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+public enum DataFormat {
+
+ SQL,
+
+ PARQUET
+}
diff --git a/src/main/java/com/github/sqljam/impexp/DbCategory.java b/src/main/java/com/github/sqljam/impexp/DbCategory.java
new file mode 100644
index 0000000..2d3bdd9
--- /dev/null
+++ b/src/main/java/com/github/sqljam/impexp/DbCategory.java
@@ -0,0 +1,44 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.impexp;
+
+/**
+ * @Description: DbCategory of databases: relational databases for transactions, OLAP databases for analytics
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+public enum DbCategory {
+
+ RELATIONAL("Relational databases"),
+
+ OLAP("OLAP databases");
+
+ private final String displayName;
+
+ DbCategory(String displayName) {
+ this.displayName = displayName;
+ }
+
+ public String getDisplayName() {
+ return displayName;
+ }
+
+ @Override
+ public String toString() {
+ return displayName;
+ }
+}
diff --git a/src/main/java/com/github/sqljam/impexp/DbType.java b/src/main/java/com/github/sqljam/impexp/DbType.java
index 8234ef1..ad82908 100644
--- a/src/main/java/com/github/sqljam/impexp/DbType.java
+++ b/src/main/java/com/github/sqljam/impexp/DbType.java
@@ -20,13 +20,18 @@
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.Validate;
-import com.github.sqljam.utils.CaseInsensitiveMap;
+import com.github.sqljam.impexp.db.ClickHouseDialect;
+import com.github.sqljam.impexp.db.ClickHouseMetaDataOperations;
+import com.github.sqljam.impexp.db.DuckDBDialect;
+import com.github.sqljam.impexp.db.DuckDBMetaDataOperations;
import com.github.sqljam.impexp.db.H2Dialect;
import com.github.sqljam.impexp.db.H2MetaDataOperations;
-import com.github.sqljam.impexp.db.MySQLDialect;
+import com.github.sqljam.impexp.db.MariaDBDialect;
+import com.github.sqljam.impexp.db.MariaDBMetaDataOperations;
import com.github.sqljam.impexp.db.MySQL55Dialect;
import com.github.sqljam.impexp.db.MySQL56Dialect;
import com.github.sqljam.impexp.db.MySQL57Dialect;
+import com.github.sqljam.impexp.db.MySQLDialect;
import com.github.sqljam.impexp.db.MySQLMetaDataOperations;
import com.github.sqljam.impexp.db.Oracle11gDialect;
import com.github.sqljam.impexp.db.Oracle12cDialect;
@@ -41,6 +46,7 @@
import com.github.sqljam.impexp.db.SQLServerMetaDataOperations;
import com.github.sqljam.impexp.db.SQLiteDialect;
import com.github.sqljam.impexp.db.SQLiteMetaDataOperations;
+import com.github.sqljam.utils.CaseInsensitiveMap;
/**
* @Description: DbType is a supported database type, it creates dialects (by versions) and metadata operations
@@ -77,6 +83,24 @@ public String getUrl(String hostname, int port, String catalogName) {
}
},
+ MARIADB("MariaDB", "org.mariadb.jdbc.Driver", 3306, true, false, false, MariaDBDialect::new,
+ MariaDBMetaDataOperations::new) {
+ /**
+ * Sequences of MariaDB 10.3+ are supported by the dialect, versions are not subclassed
+ */
+ @Override
+ protected Dialect createVersionDialect(int majorVersion, int minorVersion) {
+ return new MariaDBDialect();
+ }
+
+ @Override
+ public String getUrl(String hostname, int port, String catalogName) {
+ String database = StringUtils.isNotBlank(catalogName) ? "/" + catalogName : "";
+ // YEAR is a number, as YEAR of MySQL
+ return String.format("jdbc:mariadb://%s:%d%s?yearIsDateType=false", hostname, port, database);
+ }
+ },
+
POSTGRESQL("PostgreSQL", "org.postgresql.Driver", 5432, false, true, true, PostgreSQLDialect::new,
PostgreSQLMetaDataOperations::new) {
@Override
@@ -175,6 +199,61 @@ public boolean isCatalogSupported() {
public boolean isFileBased() {
return true;
}
+ },
+
+ DUCKDB("DuckDB", "org.duckdb.DuckDBDriver", 0, false, true, true, DuckDBDialect::new,
+ DuckDBMetaDataOperations::new) {
+ @Override
+ public String getUrl(String hostname, int port, String catalogName) {
+ Validate.notBlank(catalogName, "Database file must be required while connecting to duckdb database.");
+ return "jdbc:duckdb:" + catalogName;
+ }
+
+ @Override
+ public boolean isCatalogSupported() {
+ return false;
+ }
+
+ @Override
+ public boolean isFileBased() {
+ return true;
+ }
+
+ @Override
+ public DbCategory getCategory() {
+ return DbCategory.OLAP;
+ }
+ },
+
+ CLICKHOUSE("ClickHouse", "com.clickhouse.jdbc.ClickHouseDriver", 8123, false, true, true, ClickHouseDialect::new,
+ ClickHouseMetaDataOperations::new) {
+ @Override
+ public String getUrl(String hostname, int port, String catalogName) {
+ String database = StringUtils.isNotBlank(catalogName) ? "/" + catalogName : "";
+ return normalizeUrl(String.format("jdbc:clickhouse:http://%s:%d%s", hostname, port, database));
+ }
+
+ /**
+ * Compression of ClickHouse responses is turned off unless it is configured, the LZ4 frames of recent
+ * servers are not read by the driver
+ */
+ @Override
+ public String normalizeUrl(String url) {
+ if (StringUtils.isBlank(url) || StringUtils.containsIgnoreCase(url, "compress=")) {
+ return url;
+ }
+ return url + (url.contains("?") ? "&" : "?") + "compress=0";
+ }
+
+ @Override
+ public boolean isCatalogSupported() {
+ return false;
+ }
+
+ @Override
+ public DbCategory getCategory() {
+ return DbCategory.OLAP;
+ }
};
private DbType(String displayName, String driverClassName, int defaultPort, boolean canSetCatalog,
@@ -230,6 +309,13 @@ public boolean isCatalogSupported() {
return true;
}
+ /**
+ * Relational database (default) or OLAP database
+ */
+ public DbCategory getCategory() {
+ return DbCategory.RELATIONAL;
+ }
+
/**
* Whether the database is addressed by a file path instead of host and port
*/
@@ -268,6 +354,13 @@ public MetaDataOperations createMetaDataOperations() {
public abstract String getUrl(String hostname, int port, String catalogName);
+ /**
+ * Jdbc url with the options required by SqlJam, e.g. a url entered by the user
+ */
+ public String normalizeUrl(String url) {
+ return url;
+ }
+
@Override
public String toString() {
return displayName;
@@ -290,8 +383,10 @@ public static DbType forUrl(String jdbcUrl) {
return null;
}
String url = jdbcUrl.toLowerCase();
- if (url.startsWith("jdbc:mysql:") || url.startsWith("jdbc:mariadb:")) {
+ if (url.startsWith("jdbc:mysql:")) {
return MYSQL;
+ } else if (url.startsWith("jdbc:mariadb:")) {
+ return MARIADB;
} else if (url.startsWith("jdbc:postgresql:")) {
return POSTGRESQL;
} else if (url.startsWith("jdbc:oracle:")) {
@@ -302,6 +397,10 @@ public static DbType forUrl(String jdbcUrl) {
return H2;
} else if (url.startsWith("jdbc:sqlite:")) {
return SQLITE;
+ } else if (url.startsWith("jdbc:duckdb:")) {
+ return DUCKDB;
+ } else if (url.startsWith("jdbc:clickhouse:") || url.startsWith("jdbc:ch:")) {
+ return CLICKHOUSE;
}
return null;
}
diff --git a/src/main/java/com/github/sqljam/impexp/DdlScripter.java b/src/main/java/com/github/sqljam/impexp/DdlScripter.java
index 8f74e98..269e777 100644
--- a/src/main/java/com/github/sqljam/impexp/DdlScripter.java
+++ b/src/main/java/com/github/sqljam/impexp/DdlScripter.java
@@ -19,10 +19,11 @@
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import lombok.Getter;
-import lombok.Setter;
+
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.utils.MapUtils;
+import lombok.Getter;
+import lombok.Setter;
/**
* @Description: DdlScripter holds statements of catalogs, schemas and tables, and formats them as pretty scripts or plain statements
diff --git a/src/main/java/com/github/sqljam/impexp/DefaultMetaDataVisitor.java b/src/main/java/com/github/sqljam/impexp/DefaultMetaDataVisitor.java
index f5df924..66ca6cb 100644
--- a/src/main/java/com/github/sqljam/impexp/DefaultMetaDataVisitor.java
+++ b/src/main/java/com/github/sqljam/impexp/DefaultMetaDataVisitor.java
@@ -94,15 +94,23 @@ public void visit(TableMetaData metaData) throws SQLException {
metaData.getTableName());
addStatements(table.getCreateTableStatements(), statements);
String tableOptions = configuration.getDialect().getTableOptions(metaData.getCatalogName(),
- metaData.getSchemaName(), metaData.getTableName(), metaData.getDetail());
+ metaData.getSchemaName(), configuration.getDialect().getTargetTableName(metaData.getTableName()),
+ metaData.getDetail(),
+ metaData.getPrimaryKeyColumnNames());
if (StringUtils.isNotBlank(tableOptions)) {
table.getTableOptions().add(tableOptions);
}
if (configuration.isTableRecreated()) {
String dropStatement = configuration.getDialect().getDropTableStatement(metaData.getCatalogName(),
metaData.getSchemaName(),
- metaData.getTableName());
- table.getBeforeStatements().add(0, dropStatement);
+ configuration.getDialect().getTargetTableName(metaData.getTableName()));
+ if (configuration.getDialect().isForeignKeyOrderRequired()) {
+ // Tables are visited with referenced tables first, they are dropped in the reverse order
+ Schema schema = getDdlScripter().getSchema(metaData.getCatalogName(), metaData.getSchemaName());
+ schema.getBeforeStatements().add(0, dropStatement);
+ } else {
+ table.getBeforeStatements().add(0, dropStatement);
+ }
}
}
}
@@ -117,7 +125,7 @@ public void visit(PartitionTableMetaData metaData) throws SQLException {
if (configuration.isTableRecreated()) {
String dropStatement = configuration.getDialect().getDropTableStatement(metaData.getCatalogName(),
metaData.getSchemaName(),
- metaData.getTableName());
+ configuration.getDialect().getTargetTableName(metaData.getTableName()));
table.getBeforeStatements().add(0, dropStatement);
}
}
@@ -130,10 +138,19 @@ public void visit(ColumnMetaData metaData) throws SQLException {
Table table = getDdlScripter().getTable(metaData.getCatalogName(), metaData.getSchemaName(),
metaData.getTableName());
addStatements(table.getColumnStatements(), statements);
+ // User defined types are created before the tables of the schema, once
+ String userTypeStatement = metaData.getUserTypeStatement();
+ if (StringUtils.isNotBlank(userTypeStatement)) {
+ Schema schema = getDdlScripter().getSchema(metaData.getCatalogName(), metaData.getSchemaName());
+ if (!schema.getBeforeStatements().contains(userTypeStatement)) {
+ schema.getBeforeStatements().add(userTypeStatement);
+ }
+ }
if (metaData.unwrap(TableMetaData.class).getIncrementalColumnNames(configuration.getDialect())
.contains(metaData.getColumnName())) {
String[] afterStatements = configuration.getDialect().getStatementAfterIncrementalColumnCreated(
- metaData.getCatalogName(), metaData.getSchemaName(), metaData.getTableName(),
+ metaData.getCatalogName(), metaData.getSchemaName(),
+ configuration.getDialect().getTargetTableName(metaData.getTableName()),
metaData.getColumnName());
if (ArrayUtils.isNotEmpty(afterStatements)) {
addStatements(table.getAfterStatements(), afterStatements);
diff --git a/src/main/java/com/github/sqljam/impexp/Dialect.java b/src/main/java/com/github/sqljam/impexp/Dialect.java
index 17e6391..0595f5a 100644
--- a/src/main/java/com/github/sqljam/impexp/Dialect.java
+++ b/src/main/java/com/github/sqljam/impexp/Dialect.java
@@ -16,20 +16,25 @@
package com.github.sqljam.impexp;
import java.math.BigDecimal;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Time;
import java.sql.Timestamp;
import java.sql.Types;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.LocalTime;
import java.time.OffsetDateTime;
+import java.time.OffsetTime;
+import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.Arrays;
-import java.util.List;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
@@ -38,6 +43,7 @@
import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;
+import com.github.sqljam.jdbc.JdbcUtils;
import com.github.sqljam.utils.MapUtils;
/**
@@ -62,6 +68,16 @@ public abstract class Dialect {
private static final DateTimeFormatter DATE_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd");
private static final DateTimeFormatter TIME_FORMATTER = DateTimeFormatter.ofPattern("HH:mm:ss");
+ /**
+ * Placeholder of the source table name in table name patterns
+ */
+ public static final String TABLE_PLACEHOLDER = "{table}";
+
+ /**
+ * Prefix of a type name which is used as it is, e.g. user defined types of the same database type
+ */
+ public static final String VERBATIM_TYPE_PREFIX = "verbatim:";
+
private final TypeNames typeNames = new TypeNames();
private final Set reservedWords = new HashSet<>();
/**
@@ -73,6 +89,14 @@ public abstract class Dialect {
* Database type which the metadata comes from. null means the same database as this dialect.
*/
private DbType sourceDbType;
+ /**
+ * Database type which rows read by this dialect are written to, set on the dialect of the source database
+ */
+ private DbType readTargetDbType;
+ /**
+ * Name pattern of target tables, {table} is replaced with the source table name, e.g. {table}_copy
+ */
+ private String tableNamePattern;
private IdentifierCase identifierCase = IdentifierCase.AUTO;
/**
* When set, table names in DDL/DML generated by this dialect are qualified with these names
@@ -185,6 +209,56 @@ public void setSourceDbType(DbType sourceDbType) {
this.sourceDbType = sourceDbType;
}
+ public String getTableNamePattern() {
+ return tableNamePattern;
+ }
+
+ /**
+ * Copies tables with other names, e.g. {table}_copy copies tables in the same schema. Names of keys, indexes
+ * and constraints are derived from the target table names then.
+ */
+ public void setTableNamePattern(String tableNamePattern) {
+ this.tableNamePattern = tableNamePattern;
+ }
+
+ /**
+ * Whether target tables have other names than source tables
+ */
+ public boolean isTableRenamed() {
+ return StringUtils.isNotBlank(tableNamePattern) && !TABLE_PLACEHOLDER.equals(tableNamePattern.trim());
+ }
+
+ /**
+ * Name of the target table of a source table by the table name pattern
+ */
+ public String getTargetTableName(String tableName) {
+ if (!isTableRenamed() || StringUtils.isBlank(tableName)) {
+ return tableName;
+ }
+ String pattern = tableNamePattern.trim();
+ String name = pattern.contains(TABLE_PLACEHOLDER) ? pattern.replace(TABLE_PLACEHOLDER, tableName) : pattern;
+ // The name keeps the case of the source name, e.g. EMP_COPY of EMP in Oracle
+ if (tableName.equals(tableName.toUpperCase(Locale.ENGLISH))
+ && !tableName.equals(tableName.toLowerCase(Locale.ENGLISH))) {
+ return name.toUpperCase(Locale.ENGLISH);
+ } else if (tableName.equals(tableName.toLowerCase(Locale.ENGLISH))
+ && !tableName.equals(tableName.toUpperCase(Locale.ENGLISH))) {
+ return name.toLowerCase(Locale.ENGLISH);
+ }
+ return name;
+ }
+
+ public DbType getReadTargetDbType() {
+ return readTargetDbType;
+ }
+
+ /**
+ * Database type which rows read by this dialect (as the source dialect) are written to
+ */
+ public void setReadTargetDbType(DbType readTargetDbType) {
+ this.readTargetDbType = readTargetDbType;
+ }
+
public boolean isCrossDatabase() {
return sourceDbType != null && sourceDbType != getDbType();
}
@@ -251,6 +325,8 @@ public void setDatabaseVersion(int majorVersion, int minorVersion) {
public Dialect forVersion(int majorVersion, int minorVersion) {
Dialect dialect = getDbType().createDialect(majorVersion, minorVersion);
dialect.setSourceDbType(sourceDbType);
+ dialect.setReadTargetDbType(readTargetDbType);
+ dialect.setTableNamePattern(tableNamePattern);
dialect.setIdentifierCase(identifierCase);
dialect.setTargetCatalogName(targetCatalogName);
dialect.setTargetSchemaName(targetSchemaName);
@@ -446,6 +522,14 @@ public String getTableOptions(String catalog, String schema, String tableName, M
return null;
}
+ /**
+ * Options of a table with its primary key columns, e.g. the sorting key of ClickHouse MergeTree tables
+ */
+ public String getTableOptions(String catalog, String schema, String tableName, Map detail,
+ List primaryKeyColumnNames) {
+ return getTableOptions(catalog, schema, tableName, detail);
+ }
+
/**
* Creates a sequence, returns null if sequences are not supported
*/
@@ -519,7 +603,11 @@ public String getCreatePrimaryKeyStatement(String catalog, String schema, String
String pkeyName) {
if (StringUtils.isBlank(pkeyName) || pkeyName.startsWith("SYS_") || "PRIMARY".equalsIgnoreCase(pkeyName)
|| pkeyName.startsWith("PK__") || isCrossDatabase()) {
- pkeyName = getLimitedIdentifier(foldIdentifier(tableName) + "_pkey");
+ String foldedTableName = foldIdentifier(tableName);
+ // The suffix keeps the case of the table name, e.g. EMP_PKEY of Oracle
+ boolean upper = foldedTableName.equals(foldedTableName.toUpperCase(Locale.ENGLISH))
+ && !foldedTableName.equals(foldedTableName.toLowerCase(Locale.ENGLISH));
+ pkeyName = getLimitedIdentifier(foldedTableName + (upper ? "_PKEY" : "_pkey"));
return String.format("CONSTRAINT %s PRIMARY KEY (%s)", quoteIdentifier(pkeyName), getIdentifiers(columnName));
}
return String.format("CONSTRAINT %s PRIMARY KEY (%s)", getIdentifier(pkeyName), getIdentifiers(columnName));
@@ -572,6 +660,20 @@ public String getCreateForeignKeyStatement(String catalog, String schema, String
return sql.toString();
}
+ /**
+ * Whether empty strings and binaries are stored as NULL (Oracle), empty LOBs are bound as LOB objects then
+ */
+ public boolean isEmptyValueNull() {
+ return false;
+ }
+
+ /**
+ * Whether LOB values of scripts can be restored by UPDATE statements after rows are inserted
+ */
+ public boolean isLobSeparationSupported() {
+ return true;
+ }
+
/**
* Whether foreign keys must be defined in create table statement
*/
@@ -579,6 +681,21 @@ public boolean isForeignKeyInline() {
return false;
}
+ /**
+ * Whether arrays kept as text (e.g. [1, 2, 3]) are bound as java arrays to array columns
+ */
+ public boolean isArrayParameterSupported() {
+ return true;
+ }
+
+ /**
+ * Whether foreign keys are checked while creating, loading and dropping tables (DuckDB): referenced tables are
+ * created and loaded first, referencing tables are dropped first
+ */
+ public boolean isForeignKeyOrderRequired() {
+ return false;
+ }
+
protected boolean isOnUpdateRuleSupported() {
return true;
}
@@ -659,6 +776,10 @@ public String getColumnStatement(String catalog, String schema, String tableName
* Resolves column type of this dialect from jdbc metadata of the source column
*/
public String getColumnTypeName(int dataType, String typeName, int columnSize, int columnScale) {
+ if (typeName != null && typeName.startsWith(VERBATIM_TYPE_PREFIX)) {
+ // A type of the same database type, e.g. an enum type of PostgreSQL
+ return typeName.substring(VERBATIM_TYPE_PREFIX.length());
+ }
if (isCrossDatabase() && StringUtils.startsWithIgnoreCase(typeName, "interval")) {
// Interval literals are not compatible across databases
dataType = Types.VARCHAR;
@@ -1048,10 +1169,112 @@ public String getSelectColumnExpression(String columnName, String typeName) {
return quoteIdentifier(columnName);
}
+ /**
+ * Reads a column value of a select statement generated by this dialect as a portable java object
+ */
+ public Object getColumnValue(ResultSet rs, int columnIndex, int columnType, String columnTypeName)
+ throws SQLException {
+ return JdbcUtils.getColumnValue(rs, columnIndex, columnType, columnTypeName);
+ }
+
public String getCountTableStatement(String catalog, String schema, String tableName) {
return String.format("SELECT COUNT(*) FROM %s", getSourceTableName(catalog, schema, tableName));
}
+ /**
+ * Rows of a table narrowed by a query: selected columns (all columns if none), condition and groups. Grouped
+ * rows have the group columns and the count of rows.
+ *
+ * @param columnNames columns of the table, the selected ones are read
+ * @param typeNames type names of the columns
+ */
+ public String getQueryStatement(String catalog, String schema, String tableName, String[] columnNames,
+ String[] typeNames, TableQuery query) {
+ if (query == null || query.isEmpty()) {
+ return getSelectTableStatement(catalog, schema, tableName, columnNames, typeNames);
+ }
+ StringBuilder sql = new StringBuilder("SELECT ");
+ if (query.isGrouped()) {
+ sql.append(query.getGroupBy().stream().map(this::quoteIdentifier).collect(Collectors.joining(",")))
+ .append(",COUNT(*) AS ").append(getCountColumnName());
+ } else {
+ List columns = new ArrayList<>();
+ for (int i = 0; i < columnNames.length; i++) {
+ if (query.getColumns().isEmpty() || query.getColumns().contains(columnNames[i])) {
+ columns.add(getSelectColumnExpression(columnNames[i], typeNames != null ? typeNames[i] : null));
+ }
+ }
+ sql.append(String.join(",", columns));
+ }
+ sql.append(" FROM ").append(getSourceTableName(catalog, schema, tableName));
+ if (StringUtils.isNotBlank(query.getWhere())) {
+ sql.append(" WHERE ").append(query.getWhere());
+ }
+ if (query.isGrouped()) {
+ sql.append(" GROUP BY ").append(query.getGroupBy().stream().map(this::quoteIdentifier)
+ .collect(Collectors.joining(",")));
+ }
+ return sql.toString();
+ }
+
+ /**
+ * Order of rows narrowed by a query: the order of the query followed by the primary key columns which are not
+ * ordered yet (pages stay stable), the group columns of grouped rows, the primary key without query order
+ *
+ * @param keyColumnNames primary key columns, not quoted
+ * @return null if there is no order
+ */
+ public String getQueryOrderBy(TableQuery query, List keyColumnNames) {
+ if (query != null && query.isGrouped()) {
+ return StringUtils.isNotBlank(query.getOrderBy()) ? query.getOrderBy()
+ : query.getGroupBy().stream().map(this::quoteIdentifier).collect(Collectors.joining(","));
+ }
+ String keys = keyColumnNames.isEmpty() ? null
+ : keyColumnNames.stream().map(this::quoteIdentifier).collect(Collectors.joining(","));
+ if (query == null || StringUtils.isBlank(query.getOrderBy())) {
+ return keys;
+ }
+ String lowerOrderBy = query.getOrderBy().toLowerCase(Locale.ENGLISH);
+ StringBuilder order = new StringBuilder(query.getOrderBy());
+ for (String keyColumn : keyColumnNames) {
+ Pattern ordered = Pattern.compile("(^|[\\s,.`\"\\[])" + Pattern.quote(keyColumn.toLowerCase(
+ Locale.ENGLISH)) + "($|[\\s,`\"\\]])");
+ if (!ordered.matcher(lowerOrderBy).find()) {
+ order.append(",").append(quoteIdentifier(keyColumn));
+ }
+ }
+ return order.toString();
+ }
+
+ /**
+ * Name of the count column of grouped rows in the case of unquoted identifiers, so that ORDER BY row_count
+ * works without quotes
+ */
+ public String getCountColumnName() {
+ return getStoredIdentifierCase() == IdentifierCase.UPPER
+ ? TableQuery.COUNT_COLUMN.toUpperCase(Locale.ENGLISH) : TableQuery.COUNT_COLUMN;
+ }
+
+ /**
+ * Number of rows of a table narrowed by a query, groups are counted for grouped rows
+ */
+ public String getCountQueryStatement(String catalog, String schema, String tableName, TableQuery query) {
+ if (query == null || (StringUtils.isBlank(query.getWhere()) && !query.isGrouped())) {
+ return getCountTableStatement(catalog, schema, tableName);
+ }
+ String count = getCountTableStatement(catalog, schema, tableName);
+ if (StringUtils.isNotBlank(query.getWhere())) {
+ count += " WHERE " + query.getWhere();
+ }
+ if (!query.isGrouped()) {
+ return count;
+ }
+ String groups = query.getGroupBy().stream().map(this::quoteIdentifier).collect(Collectors.joining(","));
+ return String.format("SELECT COUNT(*) FROM (SELECT %s FROM %s%s GROUP BY %s) t", groups,
+ getSourceTableName(catalog, schema, tableName), StringUtils.isNotBlank(query.getWhere())
+ ? " WHERE " + query.getWhere() : "", groups);
+ }
+
public String getSelectMaxColumnStatement(String catalog, String schema, String tableName, String columnName) {
return String.format("SELECT max(%s) FROM %s", quoteIdentifier(columnName), getSourceTableName(catalog, schema, tableName));
}
@@ -1169,8 +1392,8 @@ public String getStringValue(String catalog, String schema, String tableName, St
return getTimestampLiteral(((Timestamp) columnValue).toLocalDateTime());
} else if (columnValue instanceof java.sql.Date) {
return getDateLiteral(((java.sql.Date) columnValue).toLocalDate());
- } else if (columnValue instanceof java.sql.Time) {
- return getTimeLiteral(((java.sql.Time) columnValue).toLocalTime());
+ } else if (columnValue instanceof Time) {
+ return getTimeLiteral(((Time) columnValue).toLocalTime());
} else if (columnValue instanceof java.util.Date) {
return getTimestampLiteral(new Timestamp(((java.util.Date) columnValue).getTime()).toLocalDateTime());
} else if (columnValue instanceof LocalDateTime) {
@@ -1183,8 +1406,8 @@ public String getStringValue(String catalog, String schema, String tableName, St
return getTimestampWithTimeZoneLiteral((OffsetDateTime) columnValue);
} else if (columnValue instanceof Object[]) {
return getArrayLiteral(catalog, schema, tableName, columnName, (Object[]) columnValue);
- } else if (columnValue instanceof java.time.OffsetTime) {
- java.time.OffsetTime time = (java.time.OffsetTime) columnValue;
+ } else if (columnValue instanceof OffsetTime) {
+ OffsetTime time = (OffsetTime) columnValue;
return getStringLiteral(formatTime(time.toLocalTime()) + time.getOffset());
} else if (columnValue instanceof ZonedDateTime) {
return getTimestampLiteral(((ZonedDateTime) columnValue).toLocalDateTime());
@@ -1219,19 +1442,20 @@ public Object getJdbcValue(Object value, int sourceDataType, String sourceTypeNa
value = ((Number) value).intValue() != 0;
}
if (isCrossDatabase()) {
- if (value instanceof CharSequence && "timetz".equalsIgnoreCase(sourceTypeName)) {
+ if (value instanceof CharSequence && ("timetz".equalsIgnoreCase(sourceTypeName)
+ || "time with time zone".equalsIgnoreCase(sourceTypeName))) {
String text = value.toString().replaceAll("([+-]\\d{2})$", "$1:00");
- value = java.time.OffsetTime.parse(text).toLocalTime();
+ value = OffsetTime.parse(text).toLocalTime();
} else if (sourceDataType == Types.BIT && columnSize > 1) {
// Bit strings are migrated as integers
value = toLong(value);
} else if (value instanceof Object[]) {
value = Arrays.deepToString((Object[]) value);
- } else if (value instanceof java.time.OffsetTime) {
- value = ((java.time.OffsetTime) value).toLocalTime();
+ } else if (value instanceof OffsetTime) {
+ value = ((OffsetTime) value).toLocalTime();
} else if (value instanceof OffsetDateTime) {
// Instant in local time zone
- value = ((OffsetDateTime) value).atZoneSameInstant(java.time.ZoneId.systemDefault())
+ value = ((OffsetDateTime) value).atZoneSameInstant(ZoneId.systemDefault())
.toLocalDateTime();
}
// Fractional seconds are truncated, rounding may overflow (e.g. 9999-12-31 23:59:59.9999999)
@@ -1244,6 +1468,41 @@ public Object getJdbcValue(Object value, int sourceDataType, String sourceTypeNa
return getJdbcValue(value);
}
+ /**
+ * Type of a column of a user defined type (enum, domain) of the same database type, the type is created in
+ * the target schema
+ */
+ public String getUserTypeName(String catalog, String schema, String typeName) {
+ String qualifier = getQualifier(getTargetCatalogName(), getTargetSchemaName(schema));
+ return StringUtils.isNotBlank(qualifier) ? qualifier + "." + getIdentifier(typeName)
+ : getIdentifier(typeName);
+ }
+
+ /**
+ * Statement creating a user defined type of the same database type if it does not exist, null if user defined
+ * types are not supported
+ *
+ * @param userType KIND (ENUM, DOMAIN), NAME and DEFINITION of the type
+ */
+ public String getCreateUserTypeStatement(String catalog, String schema, Map userType) {
+ return null;
+ }
+
+ /**
+ * Whether values are converted by the types of target columns (getTargetJdbcValue), the types are read from
+ * the target table once per table
+ */
+ public boolean isTargetTypeConversionRequired() {
+ return false;
+ }
+
+ /**
+ * Converts a value for a target column of the given type name, e.g. text of a vector for a VECTOR column
+ */
+ public Object getTargetJdbcValue(Object value, String targetTypeName) {
+ return value;
+ }
+
/**
* Sql type used to bind NULL of a column
*/
diff --git a/src/main/java/com/github/sqljam/impexp/DuckDBWorkspace.java b/src/main/java/com/github/sqljam/impexp/DuckDBWorkspace.java
new file mode 100644
index 0000000..077782c
--- /dev/null
+++ b/src/main/java/com/github/sqljam/impexp/DuckDBWorkspace.java
@@ -0,0 +1,264 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.impexp;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import com.github.sqljam.jdbc.JdbcUtils;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * @Description: DuckDBWorkspace is a temporary DuckDB database used to write and read Parquet files. Rows of other
+ * databases are copied into the workspace and written by COPY TO, Parquet files are loaded into the
+ * workspace by read_parquet and copied into other databases. The database file is deleted on close.
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+@Slf4j
+public class DuckDBWorkspace implements Closeable {
+
+ public static final String DEFAULT_SCHEMA = "main";
+
+ private final File directory;
+ private final File file;
+ private Connection connection;
+
+ /**
+ * @param parent directory of the workspace, the system temporary directory if null
+ */
+ public DuckDBWorkspace(File parent) throws IOException {
+ File base = parent != null ? parent : FileUtils.getTempDirectory();
+ FileUtils.forceMkdir(base);
+ this.directory = Files.createTempDirectory(base.toPath(), ".sqljam-workspace-").toFile();
+ this.file = new File(directory, "workspace.duckdb");
+ }
+
+ public String getUrl() {
+ return DbType.DUCKDB.getUrl(null, 0, file.getAbsolutePath());
+ }
+
+ public File getFile() {
+ return file;
+ }
+
+ public synchronized Connection getConnection() throws SQLException {
+ if (connection == null || connection.isClosed()) {
+ connection = DriverManager.getConnection(getUrl());
+ }
+ return connection;
+ }
+
+ /**
+ * Tables of the workspace: schema and table name
+ */
+ public List getTables() throws SQLException {
+ String sql = "SELECT schema_name, table_name FROM duckdb_tables() WHERE database_name = current_database()"
+ + " AND NOT internal ORDER BY schema_name, table_name";
+ return JdbcUtils.fetchAll(getConnection(), sql, new Object[0]).stream()
+ .map(row -> new String[]{(String) row.get("schema_name"), (String) row.get("table_name")})
+ .collect(Collectors.toList());
+ }
+
+ public long countRows(String schema, String table) throws SQLException {
+ Map row = JdbcUtils.fetchOne(getConnection(),
+ String.format("SELECT count(*) AS n FROM %s", qualify(schema, table)), new Object[0]);
+ Object count = row != null ? row.get("n") : null;
+ return count instanceof Number ? ((Number) count).longValue() : 0;
+ }
+
+ /**
+ * Writes rows of the table into a Parquet file. Integers wider than DECIMAL(38) have no Parquet type, they are
+ * written as DECIMAL(38,0).
+ *
+ * @param compression SNAPPY, ZSTD, GZIP or UNCOMPRESSED
+ * @param metadata key value metadata of the file
+ */
+ public void writeParquet(String schema, String table, File parquetFile, String compression,
+ Map metadata) throws SQLException, IOException {
+ FileUtils.forceMkdirParent(parquetFile);
+ String columnSql = "SELECT column_name, data_type FROM duckdb_columns() WHERE database_name = current_database()"
+ + " AND schema_name = ? AND table_name = ? ORDER BY column_index";
+ List columns = new ArrayList<>();
+ for (Map row : JdbcUtils.fetchAll(getConnection(), columnSql, new Object[]{schema, table})) {
+ String column = quote((String) row.get("column_name"));
+ String dataType = StringUtils.upperCase((String) row.get("data_type"), Locale.ENGLISH);
+ if ("HUGEINT".equals(dataType) || "UHUGEINT".equals(dataType)) {
+ columns.add(String.format("CAST(%1$s AS DECIMAL(38,0)) AS %1$s", column));
+ } else {
+ columns.add(column);
+ }
+ }
+ StringBuilder options = new StringBuilder("FORMAT parquet, COMPRESSION ")
+ .append(StringUtils.defaultIfBlank(compression, "ZSTD").toLowerCase(Locale.ENGLISH));
+ if (metadata != null && !metadata.isEmpty()) {
+ options.append(", KV_METADATA {").append(metadata.entrySet().stream()
+ .map(e -> e.getKey() + ": " + literal(e.getValue())).collect(Collectors.joining(", ")))
+ .append("}");
+ }
+ String sql = String.format("COPY (SELECT %s FROM %s) TO %s (%s)", String.join(", ", columns),
+ qualify(schema, table), literal(parquetFile.getAbsolutePath()), options);
+ execute(sql);
+ }
+
+ /**
+ * Loads Parquet files into a table of the workspace. Identity columns get a sequence and a nextval default, so
+ * that they are recognized as identity columns of DuckDB.
+ */
+ public void loadParquet(String schema, String table, List parquetFiles, List identityColumns)
+ throws SQLException {
+ String files = parquetFiles.stream().map(f -> literal(f.getAbsolutePath()))
+ .collect(Collectors.joining(", ", "[", "]"));
+ execute(String.format("CREATE SCHEMA IF NOT EXISTS %s", quote(schema)));
+ execute(String.format("CREATE OR REPLACE TABLE %s AS SELECT * FROM read_parquet(%s)", qualify(schema, table),
+ files));
+ if (identityColumns != null) {
+ for (String column : identityColumns) {
+ String sequence = qualify(schema, table + "_" + column + "_seq");
+ execute(String.format("CREATE SEQUENCE IF NOT EXISTS %s START 1", sequence));
+ execute(String.format("ALTER TABLE %s ALTER COLUMN %s SET DEFAULT nextval('%s')",
+ qualify(schema, table), quote(column), sequence.replace("'", "''")));
+ }
+ }
+ }
+
+ /**
+ * Drops columns of a workspace table, e.g. columns which are generated by the target database
+ */
+ public void dropColumns(String schema, String table, List columns) throws SQLException {
+ for (String column : columns) {
+ execute(String.format("ALTER TABLE %s DROP COLUMN IF EXISTS %s", qualify(schema, table), quote(column)));
+ }
+ }
+
+ /**
+ * Columns of a workspace table: name and DuckDB type
+ */
+ public List describeTable(String schema, String table) throws SQLException {
+ List columns = new ArrayList<>();
+ try (Statement statement = getConnection().createStatement();
+ ResultSet rs = statement.executeQuery(String.format("DESCRIBE %s", qualify(schema, table)))) {
+ while (rs.next()) {
+ columns.add(new String[]{rs.getString("column_name"), rs.getString("column_type")});
+ }
+ }
+ return columns;
+ }
+
+ /**
+ * Columns of Parquet files: name and DuckDB type
+ */
+ public List describeParquet(List parquetFiles) throws SQLException {
+ String files = parquetFiles.stream().map(f -> literal(f.getAbsolutePath()))
+ .collect(Collectors.joining(", ", "[", "]"));
+ List columns = new ArrayList<>();
+ try (Statement statement = getConnection().createStatement();
+ ResultSet rs = statement.executeQuery(String.format("DESCRIBE SELECT * FROM read_parquet(%s)", files))) {
+ while (rs.next()) {
+ columns.add(new String[]{rs.getString("column_name"), rs.getString("column_type")});
+ }
+ }
+ return columns;
+ }
+
+ /**
+ * First rows of Parquet files, values as text
+ */
+ public List> readParquetRows(List parquetFiles, int limit) throws SQLException {
+ String files = parquetFiles.stream().map(f -> literal(f.getAbsolutePath()))
+ .collect(Collectors.joining(", ", "[", "]"));
+ List columns = describeParquet(parquetFiles).stream()
+ .map(column -> String.format("CAST(%s AS VARCHAR)", quote(column[0])))
+ .collect(Collectors.toList());
+ List> rows = new ArrayList<>();
+ if (columns.isEmpty()) {
+ return rows;
+ }
+ String sql = String.format("SELECT %s FROM read_parquet(%s) LIMIT %d", String.join(", ", columns), files,
+ Math.max(limit, 0));
+ try (Statement statement = getConnection().createStatement(); ResultSet rs = statement.executeQuery(sql)) {
+ int count = rs.getMetaData().getColumnCount();
+ while (rs.next()) {
+ List row = new ArrayList<>(count);
+ for (int i = 1; i <= count; i++) {
+ row.add(rs.getString(i));
+ }
+ rows.add(row);
+ }
+ }
+ return rows;
+ }
+
+ public long countParquetRows(List parquetFiles) throws SQLException {
+ String files = parquetFiles.stream().map(f -> literal(f.getAbsolutePath()))
+ .collect(Collectors.joining(", ", "[", "]"));
+ Map row = JdbcUtils.fetchOne(getConnection(),
+ String.format("SELECT count(*) AS n FROM read_parquet(%s)", files), new Object[0]);
+ Object count = row != null ? row.get("n") : null;
+ return count instanceof Number ? ((Number) count).longValue() : 0;
+ }
+
+ public void execute(String sql) throws SQLException {
+ if (log.isDebugEnabled()) {
+ log.debug("Workspace: {}", sql);
+ }
+ try (Statement statement = getConnection().createStatement()) {
+ statement.execute(sql);
+ }
+ }
+
+ public static String quote(String name) {
+ return "\"" + name.replace("\"", "\"\"") + "\"";
+ }
+
+ public static String qualify(String schema, String table) {
+ return StringUtils.isNotBlank(schema) ? quote(schema) + "." + quote(table) : quote(table);
+ }
+
+ private static String literal(String value) {
+ return "'" + value.replace("'", "''") + "'";
+ }
+
+ /**
+ * Closes the connection of the workspace, it is opened again when needed
+ */
+ public synchronized void closeConnection() {
+ JdbcUtils.closeQuietly(connection);
+ connection = null;
+ }
+
+ @Override
+ public synchronized void close() {
+ JdbcUtils.closeQuietly(connection);
+ connection = null;
+ FileUtils.deleteQuietly(directory);
+ }
+}
diff --git a/src/main/java/com/github/sqljam/impexp/ExportManifest.java b/src/main/java/com/github/sqljam/impexp/ExportManifest.java
index b800f8a..901fca4 100644
--- a/src/main/java/com/github/sqljam/impexp/ExportManifest.java
+++ b/src/main/java/com/github/sqljam/impexp/ExportManifest.java
@@ -67,7 +67,12 @@ public enum FileType {
LOB_MANIFEST,
- CONSTRAINTS
+ CONSTRAINTS,
+
+ /**
+ * Rows of a table in a Parquet file
+ */
+ PARQUET
}
private String format = FORMAT;
@@ -75,6 +80,10 @@ public enum FileType {
private Status status;
private String createdAt = LocalDateTime.now().toString();
private ExportMode exportMode;
+ /**
+ * Format of rows, SQL for packages written before Parquet was supported
+ */
+ private DataFormat dataFormat = DataFormat.SQL;
private Database source = new Database();
private Database target = new Database();
private Map options = new LinkedHashMap<>();
@@ -125,6 +134,10 @@ public static class TableEntry {
private long rows;
private Set dataFiles = new LinkedHashSet<>();
private long lobFiles;
+ /**
+ * Identity columns of the table (Parquet packages), their values are kept and sequences are reset
+ */
+ private List identityColumns = new ArrayList<>();
}
private static ObjectMapper createObjectMapper() {
diff --git a/src/main/java/com/github/sqljam/impexp/Exporter.java b/src/main/java/com/github/sqljam/impexp/Exporter.java
index 28c191a..fad6d91 100644
--- a/src/main/java/com/github/sqljam/impexp/Exporter.java
+++ b/src/main/java/com/github/sqljam/impexp/Exporter.java
@@ -18,13 +18,13 @@
import java.sql.Connection;
import java.sql.DatabaseMetaData;
import java.sql.SQLException;
-import java.util.List;
+import java.util.HashMap;
import java.util.Map;
-import java.util.stream.Collectors;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.jdbc.ConnectionFactory;
+import com.github.sqljam.jdbc.JdbcUtils;
import com.github.sqljam.jdbc.SimpleConnectionFactory;
import com.github.sqljam.jdbc.page.MapBasedPageReader;
import com.github.sqljam.page.EachPage;
@@ -110,6 +110,22 @@ public static class ExportConfiguration {
* Dialect of source database to read data. Default is the dialect of dbType.
*/
private Dialect sourceDialect;
+ /**
+ * Queries narrowing the rows and columns of tables by table name, e.g. rows filtered in the data viewer
+ */
+ private Map tableQueries = new HashMap<>();
+
+ /**
+ * Query of a table, null if all rows and columns are exported. Grouped rows are not rows of the table.
+ */
+ public TableQuery getTableQuery(String tableName) {
+ TableQuery query = tableQueries != null ? tableQueries.get(tableName) : null;
+ if (query != null && query.isGrouped()) {
+ throw new ImpExpException("Grouped rows of table " + tableName + " can not be exported,"
+ + " the rows are exported without GROUP BY");
+ }
+ return query == null || query.isEmpty() ? null : query;
+ }
public Dialect getDialect() {
if (dialect == null && dbType != null) {
@@ -216,7 +232,7 @@ public void export(ExportMode exportMode) throws Exception {
}
}
- private final Map tableRows = new java.util.HashMap<>();
+ private final Map tableRows = new HashMap<>();
private long totalRows;
private long processedRows;
@@ -237,9 +253,10 @@ private void countRows(ServerMetaData serverMetaData, ConnectionFactory connecti
continue;
}
checkCancelled();
- String sql = sourceDialect.getCountTableStatement(catalogMd.getCatalogName(),
- schemaMd.getSchemaName(), tableMd.getTableName());
- Long rows = com.github.sqljam.jdbc.JdbcUtils.fetchOne(connection, sql, Long.class);
+ String sql = sourceDialect.getCountQueryStatement(catalogMd.getCatalogName(),
+ schemaMd.getSchemaName(), tableMd.getTableName(),
+ configuration.getTableQuery(tableMd.getTableName()));
+ Long rows = JdbcUtils.fetchOne(connection, sql, Long.class);
tableRows.put(tableMd, rows != null ? rows : 0L);
totalRows += rows != null ? rows : 0L;
}
@@ -324,16 +341,18 @@ private void scanTable(String catalogName, String schemaName, String tableName,
ConnectionFactory connectionFactory) {
checkCancelled();
Dialect sourceDialect = configuration.getSourceDialect();
+ if (configuration.getDialect() != null) {
+ sourceDialect.setReadTargetDbType(configuration.getDialect().getDbType());
+ }
String[] columnNames = tableMetaData.getColumnMetaDatas().stream().map(ColumnMetaData::getColumnName)
.toArray(String[]::new);
String[] typeNames = tableMetaData.getColumnMetaDatas().stream()
.map(md -> (String) md.getDetail().get("TYPE_NAME")).toArray(String[]::new);
+ TableQuery query = configuration.getTableQuery(tableName);
String sql = columnNames.length > 0
- ? sourceDialect.getSelectTableStatement(catalogName, schemaName, tableName, columnNames, typeNames)
+ ? sourceDialect.getQueryStatement(catalogName, schemaName, tableName, columnNames, typeNames, query)
: sourceDialect.getSelectTableStatement(catalogName, schemaName, tableName);
- List orderColumns = tableMetaData.getPrimaryKeyColumnNames();
- String orderBy = orderColumns.isEmpty() ? null
- : orderColumns.stream().map(sourceDialect::quoteIdentifier).collect(Collectors.joining(","));
+ String orderBy = sourceDialect.getQueryOrderBy(query, tableMetaData.getPrimaryKeyColumnNames());
// Rows were counted already
long rows = tableRows.getOrDefault(tableMetaData, -1L);
PageReader> pageReader = new MapBasedPageReader(connectionFactory, sql, new Object[0],
diff --git a/src/main/java/com/github/sqljam/impexp/ForeignKeyMetaData.java b/src/main/java/com/github/sqljam/impexp/ForeignKeyMetaData.java
index 3850907..c5c6f42 100644
--- a/src/main/java/com/github/sqljam/impexp/ForeignKeyMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/ForeignKeyMetaData.java
@@ -48,8 +48,18 @@ public String[] getStatements() throws SQLException {
String[] columnNames = details.stream().map(info -> (String) info.get("FKCOLUMN_NAME")).toArray(String[]::new);
String[] refColumnNames = details.stream().map(info -> (String) info.get("PKCOLUMN_NAME"))
.toArray(String[]::new);
- String statement = getDialect().getCreateForeignKeyStatement(getCatalogName(), getSchemaName(), getTableName(),
- foreignKeyName, columnNames, getReferencedTableName(), refColumnNames,
+ Dialect dialect = getDialect();
+ String refTableName = getReferencedTableName();
+ String fkName = foreignKeyName;
+ if (dialect.isTableRenamed()) {
+ // Copies reference copies of tables which are copied together, other tables are referenced as they are
+ if (unwrap(SchemaMetaData.class).findTableMetaData(refTableName).isPresent()) {
+ refTableName = dialect.getTargetTableName(refTableName);
+ }
+ fkName = dialect.getTargetTableName(foreignKeyName);
+ }
+ String statement = dialect.getCreateForeignKeyStatement(getCatalogName(), getSchemaName(),
+ dialect.getTargetTableName(getTableName()), fkName, columnNames, refTableName, refColumnNames,
getRuleName(getDetail().get("UPDATE_RULE")), getRuleName(getDetail().get("DELETE_RULE")));
return new String[]{statement};
}
diff --git a/src/main/java/com/github/sqljam/impexp/HikariDataSourceConnectionFactory.java b/src/main/java/com/github/sqljam/impexp/HikariDataSourceConnectionFactory.java
index 7b83620..cd338b9 100644
--- a/src/main/java/com/github/sqljam/impexp/HikariDataSourceConnectionFactory.java
+++ b/src/main/java/com/github/sqljam/impexp/HikariDataSourceConnectionFactory.java
@@ -17,8 +17,9 @@
import java.sql.Connection;
import java.sql.SQLException;
-import com.github.sqljam.jdbc.ConnectionFactory;
+
import com.github.sqljam.config.Config;
+import com.github.sqljam.jdbc.ConnectionFactory;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
diff --git a/src/main/java/com/github/sqljam/impexp/ImportExportHandler.java b/src/main/java/com/github/sqljam/impexp/ImportExportHandler.java
index 933afca..8684143 100644
--- a/src/main/java/com/github/sqljam/impexp/ImportExportHandler.java
+++ b/src/main/java/com/github/sqljam/impexp/ImportExportHandler.java
@@ -18,23 +18,30 @@
import java.sql.Connection;
import java.sql.DatabaseMetaData;
import java.sql.PreparedStatement;
+import java.sql.ResultSet;
import java.sql.SQLException;
+import java.sql.Statement;
import java.sql.Types;
import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
+import com.github.sqljam.impexp.DdlScripter.Catalog;
+import com.github.sqljam.impexp.DdlScripter.Schema;
import com.github.sqljam.jdbc.ConnectionFactory;
import com.github.sqljam.jdbc.JdbcUtils;
import com.github.sqljam.jdbc.SimpleConnectionFactory;
import com.github.sqljam.page.EachPage;
-import com.github.sqljam.impexp.DdlScripter.Catalog;
-import com.github.sqljam.impexp.DdlScripter.Schema;
-
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
@@ -78,18 +85,32 @@ public static class ImportConfiguration {
* Creates target schema (database of MySQL) if it does not exist
*/
private boolean targetSchemaCreated = true;
+ /**
+ * Name pattern of target tables, {table} is replaced with the source table name, e.g. {table}_copy copies
+ * tables in the same schema. Target tables have the same names by default.
+ */
+ private String tableNamePattern;
}
private ConnectionFactory connectionFactory;
private Dialect dialect;
private ExportListener exportListener = ExportListener.NONE;
private final IdentityValueTracker identityValueTracker = new IdentityValueTracker();
+ private final Map> targetColumnTypes = new ConcurrentHashMap<>();
@Override
public void setExportListener(ExportListener exportListener) {
this.exportListener = exportListener != null ? exportListener : ExportListener.NONE;
}
+ /**
+ * Connections of the target database, e.g. an existing connection shared by {@link com.github.sqljam.jdbc.SharedConnectionFactory}.
+ * By default a connection pool is created from the configuration.
+ */
+ public void setConnectionFactory(ConnectionFactory connectionFactory) {
+ this.connectionFactory = connectionFactory;
+ }
+
@Override
public void start() {
if (connectionFactory == null) {
@@ -185,7 +206,7 @@ public void exportDdl(DdlScripter ddlScripter) throws Exception {
}
private void createTargetSchemas(DdlScripter ddlScripter) throws SQLException {
- java.util.Set schemaNames = new java.util.LinkedHashSet<>();
+ Set schemaNames = new LinkedHashSet<>();
if (!configuration.getDbType().isSchemaSupported()) {
if (StringUtils.isNotBlank(configuration.getTargetCatalogName())) {
schemaNames.add(configuration.getTargetCatalogName());
@@ -303,42 +324,73 @@ public void exportData(String catalogName,
for (int i = 0; i < columns.length; i++) {
nullTypes[i] = dialect.getNullSqlType(getNullType(tableMetaData.findColumnMetaData(columns[i]).orElse(null)));
}
- String insertSql = dialect.getInsertTableStatement(catalogName, schemaName, tableName, columns);
+ // Empty LOBs of databases which store empty values as NULL (Oracle) are bound as LOB objects
+ int[] emptyLobTypes = new int[columns.length];
+ if (dialect.isEmptyValueNull()) {
+ for (int i = 0; i < columns.length; i++) {
+ emptyLobTypes[i] = getLobType(dialect.getRegisteredColumnTypeName(tableName, columns[i]),
+ tableMetaData.findColumnMetaData(columns[i]).orElse(null));
+ }
+ }
+ String targetTableName = dialect.getTargetTableName(tableName);
+ Map targetTypes = Collections.emptyMap();
+ String insertSql = dialect.getInsertTableStatement(catalogName, schemaName, targetTableName, columns);
Connection connection = null;
boolean autoCommit = true;
+ boolean transactional = false;
try {
connection = getConnection(catalogName, schemaName);
autoCommit = connection.getAutoCommit();
- connection.setAutoCommit(false);
- executeQuietly(connection, dialect.getStatementBeforeInsert(catalogName, schemaName, tableName,
+ transactional = JdbcUtils.beginTransaction(connection);
+ if (dialect.isTargetTypeConversionRequired()) {
+ targetTypes = getTargetColumnTypes(connection, targetTableName);
+ }
+ executeQuietly(connection, dialect.getStatementBeforeInsert(catalogName, schemaName, targetTableName,
identityIncluded));
int rows = 0;
try (PreparedStatement ps = connection.prepareStatement(insertSql)) {
+ Map arrayParameters = new HashMap<>();
for (Map row : rowList) {
int index = 0;
for (Object value : row.values()) {
if (value == null) {
ps.setNull(index + 1, nullTypes[index]);
} else {
- ps.setObject(index + 1, value);
+ if (isArrayText(value) && dialect.isArrayParameterSupported()
+ && isArrayParameter(ps, index + 1, arrayParameters)) {
+ // Arrays kept as text (Parquet packages) are bound as arrays
+ value = parseArrayText(value.toString());
+ }
+ if (emptyLobTypes[index] == Types.BLOB && value instanceof byte[]
+ && ((byte[]) value).length == 0) {
+ ps.setBlob(index + 1, connection.createBlob());
+ } else if (emptyLobTypes[index] == Types.CLOB && "".equals(value)) {
+ ps.setClob(index + 1, connection.createClob());
+ } else {
+ ps.setObject(index + 1, targetTypes.isEmpty() ? value
+ : dialect.getTargetJdbcValue(value, targetTypes.get(columns[index]
+ .toLowerCase(Locale.ENGLISH))));
+ }
}
index++;
}
ps.addBatch();
}
for (int n : ps.executeBatch()) {
- rows += n > 0 ? n : (n == java.sql.Statement.SUCCESS_NO_INFO ? 1 : 0);
+ rows += n > 0 ? n : (n == Statement.SUCCESS_NO_INFO ? 1 : 0);
}
}
- executeQuietly(connection, dialect.getStatementAfterInsert(catalogName, schemaName, tableName,
+ executeQuietly(connection, dialect.getStatementAfterInsert(catalogName, schemaName, targetTableName,
identityIncluded));
- connection.commit();
+ if (transactional) {
+ connection.commit();
+ }
if (log.isInfoEnabled()) {
log.info("Execute dml: {}", insertSql);
log.info("Add {} rows to table: {}", rows, tableName);
}
} catch (Exception e) {
- if (connection != null) {
+ if (connection != null && transactional) {
try {
connection.rollback();
} catch (SQLException ignored) {
@@ -349,7 +401,7 @@ public void exportData(String catalogName,
}
throw e;
} finally {
- if (connection != null) {
+ if (connection != null && transactional) {
try {
connection.setAutoCommit(autoCommit);
} catch (SQLException ignored) {
@@ -371,8 +423,9 @@ private void resetIdentityValues(String catalogName, String schemaName, String t
}
List sqls = new ArrayList<>();
for (Map.Entry entry : maxValues.entrySet()) {
- String sql = tableMetaData.getDialect().getResetIdentityStatement(catalogName, schemaName, tableName,
- entry.getKey(), entry.getValue() + 1);
+ Dialect targetDialect = tableMetaData.getDialect();
+ String sql = targetDialect.getResetIdentityStatement(catalogName, schemaName,
+ targetDialect.getTargetTableName(tableName), entry.getKey(), entry.getValue() + 1);
if (StringUtils.isNotBlank(sql)) {
sqls.add(sql);
}
@@ -398,6 +451,42 @@ private void executeQuietly(Connection connection, String[] sqls) {
/**
* Sql type to bind NULL, some drivers (SQL Server) can not convert untyped NULL to binary columns
*/
+ /**
+ * BLOB or CLOB if the target column is a LOB, 0 otherwise. The type registered by created tables is preferred,
+ * the type of the source column is used when tables are not created.
+ */
+ /**
+ * Type names of the columns of a target table by lower case column name, read once per table
+ */
+ private Map getTargetColumnTypes(Connection connection, String targetTableName)
+ throws SQLException {
+ Map types = targetColumnTypes.get(targetTableName);
+ if (types != null) {
+ return types;
+ }
+ types = new HashMap<>();
+ try (ResultSet rs = connection.getMetaData().getColumns(connection.getCatalog(),
+ connection.getSchema(), dialect.foldIdentifier(targetTableName), null)) {
+ while (rs.next()) {
+ types.put(rs.getString("COLUMN_NAME").toLowerCase(Locale.ENGLISH),
+ rs.getString("TYPE_NAME"));
+ }
+ }
+ targetColumnTypes.put(targetTableName, types);
+ return types;
+ }
+
+ static int getLobType(String targetTypeName, ColumnMetaData columnMetaData) {
+ String typeName = StringUtils.defaultString(targetTypeName).toLowerCase(Locale.ENGLISH);
+ if (typeName.isEmpty() && columnMetaData != null) {
+ int dataType = TableMetaData.getInt(columnMetaData.getDetail(), "DATA_TYPE");
+ return dataType == Types.BLOB ? Types.BLOB : dataType == Types.CLOB || dataType == Types.NCLOB
+ ? Types.CLOB : 0;
+ }
+ return typeName.equals("blob") ? Types.BLOB : typeName.equals("clob") || typeName.equals("nclob")
+ ? Types.CLOB : 0;
+ }
+
static int getNullType(ColumnMetaData columnMetaData) {
if (columnMetaData == null) {
return Types.VARCHAR;
@@ -435,6 +524,48 @@ static int getNullType(ColumnMetaData columnMetaData) {
}
}
+ static boolean isArrayText(Object value) {
+ if (!(value instanceof CharSequence)) {
+ return false;
+ }
+ String text = value.toString();
+ return text.length() >= 2 && text.charAt(0) == '[' && text.charAt(text.length() - 1) == ']';
+ }
+
+ /**
+ * Whether the parameter is bound to an array column, the type is read once from parameter metadata
+ */
+ private static boolean isArrayParameter(PreparedStatement ps, int parameterIndex,
+ Map arrayParameters) {
+ return arrayParameters.computeIfAbsent(parameterIndex, index -> {
+ try {
+ return ps.getParameterMetaData().getParameterType(index) == Types.ARRAY;
+ } catch (SQLException | RuntimeException e) {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Elements of an array text, e.g. [1, 2, 3] or [a, b], NULL elements are nulls
+ */
+ static Object[] parseArrayText(String text) {
+ String content = text.substring(1, text.length() - 1).trim();
+ if (content.isEmpty()) {
+ return new Object[0];
+ }
+ String[] items = content.split(",\\s*");
+ Object[] elements = new Object[items.length];
+ for (int i = 0; i < items.length; i++) {
+ String item = items[i].trim();
+ if (item.length() >= 2 && item.startsWith("'") && item.endsWith("'")) {
+ item = item.substring(1, item.length() - 1).replace("''", "'");
+ }
+ elements[i] = "NULL".equalsIgnoreCase(item) ? null : item;
+ }
+ return elements;
+ }
+
protected boolean shouldFilterColumn(String columnName, TableMetaData tableMetaData) {
return tableMetaData.isAutoIncrementColumn(columnName);
}
diff --git a/src/main/java/com/github/sqljam/impexp/ImportExporter.java b/src/main/java/com/github/sqljam/impexp/ImportExporter.java
index 5cac4f6..92d69d9 100644
--- a/src/main/java/com/github/sqljam/impexp/ImportExporter.java
+++ b/src/main/java/com/github/sqljam/impexp/ImportExporter.java
@@ -17,6 +17,7 @@
import org.apache.commons.lang3.StringUtils;
import com.github.sqljam.impexp.ImportExportHandler.ImportConfiguration;
+import com.github.sqljam.jdbc.ConnectionFactory;
/**
* @Description: ImportExporter copies tables from a source database into a target database
@@ -32,6 +33,8 @@ public ImportExporter() {
private final ImportExportHandler importingExportHandler = new ImportExportHandler();
private final Exporter exporter = new Exporter(importingExportHandler);
private IdentifierCase identifierCase;
+ private Dialect targetDialect;
+ private Boolean sourceSchemaPreserved;
private int targetMajorVersion = -1;
private int targetMinorVersion = -1;
@@ -43,10 +46,33 @@ public void setMetaDataOperations(MetaDataOperations metaDataOperations) {
exporter.setMetaDataOperations(metaDataOperations);
}
+ /**
+ * Imports by connections of the factory instead of creating a connection pool, e.g. an existing connection
+ * shared by {@link com.github.sqljam.jdbc.SharedConnectionFactory}
+ */
+ public void setTargetConnectionFactory(ConnectionFactory connectionFactory) {
+ importingExportHandler.setConnectionFactory(connectionFactory);
+ }
+
public void setExportListener(ExportListener exportListener) {
exporter.setExportListener(exportListener);
}
+ /**
+ * Whether tables are imported into schemas with the same names as source schemas when no target schema is
+ * given. By default schemas are preserved if both databases support schemas.
+ */
+ public void setSourceSchemaPreserved(Boolean sourceSchemaPreserved) {
+ this.sourceSchemaPreserved = sourceSchemaPreserved;
+ }
+
+ /**
+ * Dialect of the target database with custom settings, created from the target database type by default
+ */
+ public void setTargetDialect(Dialect targetDialect) {
+ this.targetDialect = targetDialect;
+ }
+
public void setIdentifierCase(IdentifierCase identifierCase) {
this.identifierCase = identifierCase;
}
@@ -76,17 +102,25 @@ private void prepare() {
if (importConfiguration.getDbType() == null) {
importConfiguration.setDbType(DbType.forUrl(importConfiguration.getUrl()));
}
- Dialect dialect = importConfiguration.getDbType().createDialect();
+ Dialect dialect = targetDialect != null ? targetDialect : importConfiguration.getDbType().createDialect();
dialect.setSourceDbType(exportConfiguration.getDbType());
if (identifierCase != null) {
dialect.setIdentifierCase(identifierCase);
}
+ dialect.setTableNamePattern(importConfiguration.getTableNamePattern());
+ if (!dialect.isTableRenamed() && isSameTables(exportConfiguration, importConfiguration)) {
+ // Target tables would be dropped and created before rows are read from them
+ throw new ImpExpException("Source and target tables are the same, please choose another schema or a "
+ + "table name pattern such as " + Dialect.TABLE_PLACEHOLDER + "_copy");
+ }
if (StringUtils.isNotBlank(importConfiguration.getTargetSchemaName())) {
dialect.setTargetCatalogName(importConfiguration.getTargetCatalogName());
dialect.setTargetSchemaName(importConfiguration.getTargetSchemaName());
} else if (StringUtils.isNotBlank(importConfiguration.getTargetCatalogName())
&& !importConfiguration.getDbType().isSchemaSupported()) {
dialect.setTargetCatalogName(importConfiguration.getTargetCatalogName());
+ } else if (sourceSchemaPreserved != null) {
+ dialect.setSourceSchemaPreserved(sourceSchemaPreserved);
} else if (importConfiguration.getDbType().isSchemaSupported()
&& exportConfiguration.getDbType().isSchemaSupported()) {
// Schema level import: tables are imported into schemas with the same names
@@ -102,6 +136,33 @@ private void prepare() {
exportConfiguration.setShowCreateSchemaSql(false);
}
+ /**
+ * Whether rows would be imported into the tables they are read from: the same database (url) and the same
+ * catalog and schema. A target catalog or schema which is not given means the source one.
+ */
+ static boolean isSameTables(Exporter.ExportConfiguration exportConfiguration,
+ ImportConfiguration importConfiguration) {
+ if (exportConfiguration.getDbType() != importConfiguration.getDbType()
+ || StringUtils.isBlank(exportConfiguration.getUrl())
+ || !StringUtils.equals(exportConfiguration.getUrl().trim(), StringUtils.trim(importConfiguration.getUrl()))) {
+ return false;
+ }
+ return isSameName(exportConfiguration.getIncludedCatalogNames(), importConfiguration.getTargetCatalogName())
+ && isSameName(exportConfiguration.getIncludedSchemaNames(), importConfiguration.getTargetSchemaName());
+ }
+
+ private static boolean isSameName(String[] sourceNames, String targetName) {
+ if (StringUtils.isBlank(targetName) || sourceNames == null || sourceNames.length == 0) {
+ return true;
+ }
+ for (String sourceName : sourceNames) {
+ if (targetName.trim().equalsIgnoreCase(StringUtils.trim(sourceName))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
public void export(ExportMode exportMode) throws Exception {
prepare();
exporter.export(exportMode);
diff --git a/src/main/java/com/github/sqljam/impexp/IndexMetaData.java b/src/main/java/com/github/sqljam/impexp/IndexMetaData.java
index 7db79df..89276ab 100644
--- a/src/main/java/com/github/sqljam/impexp/IndexMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/IndexMetaData.java
@@ -121,7 +121,7 @@ public String[] getStatements() throws SQLException {
String catalogName = getCatalogName();
String schemaName = getSchemaName();
- String tableName = getTableName();
+ String tableName = getDialect().getTargetTableName(getTableName());
boolean partition = tableMetaData.isPartitionTable();
int typeIndex = TableMetaData.getInt(detail, "TYPE");
String indexType = typeIndex == DatabaseMetaData.tableIndexHashed ? "HASH" : null;
diff --git a/src/main/java/com/github/sqljam/impexp/ParquetExporter.java b/src/main/java/com/github/sqljam/impexp/ParquetExporter.java
new file mode 100644
index 0000000..fdab76a
--- /dev/null
+++ b/src/main/java/com/github/sqljam/impexp/ParquetExporter.java
@@ -0,0 +1,411 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.impexp;
+
+import java.io.File;
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.sql.DatabaseMetaData;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import org.apache.commons.lang3.StringUtils;
+import com.github.sqljam.impexp.db.DuckDBDialect;
+import com.github.sqljam.impexp.db.DuckDBMetaDataOperations;
+import com.github.sqljam.jdbc.SharedConnectionFactory;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * @Description: ParquetExporter exports tables as an export package with Parquet files: schema.sql and
+ * constraints.sql for the target database (the same as SQL packages), data/<table>.parquet for
+ * rows and manifest.json. Rows are copied into a temporary DuckDB workspace first (values which
+ * Parquet can not hold exactly are kept as text), then written by DuckDB.
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+@Slf4j
+public final class ParquetExporter {
+
+ public static final String PARQUET_EXTENSION = ".parquet";
+ public static final String DEFAULT_COMPRESSION = "ZSTD";
+ public static final String[] COMPRESSIONS = {"ZSTD", "SNAPPY", "GZIP", "UNCOMPRESSED"};
+
+ private final File root;
+ private final Exporter.ExportConfiguration configuration = new Exporter.ExportConfiguration();
+ private ExportListener exportListener = ExportListener.NONE;
+ private DbType targetDbType;
+ private IdentifierCase identifierCase;
+ private String targetCatalogName;
+ private String targetSchemaName;
+ private int targetMajorVersion = -1;
+ private int targetMinorVersion = -1;
+ private String compression = DEFAULT_COMPRESSION;
+ private final List writtenFiles = new ArrayList<>();
+
+ public ParquetExporter(File root) {
+ this.root = root;
+ configuration.setShowCreateUserSql(false);
+ configuration.setShowCreateCatalogSql(false);
+ configuration.setShowCreateSchemaSql(false);
+ }
+
+ /**
+ * Source database and options, the same as {@link ScriptExporter#getConfiguration()}
+ */
+ public Exporter.ExportConfiguration getConfiguration() {
+ return configuration;
+ }
+
+ public void setExportListener(ExportListener exportListener) {
+ this.exportListener = exportListener != null ? exportListener : ExportListener.NONE;
+ }
+
+ /**
+ * Database type of schema.sql and constraints.sql. Default is the same as source database.
+ */
+ public void setTargetDbType(DbType targetDbType) {
+ this.targetDbType = targetDbType;
+ }
+
+ public void setIdentifierCase(IdentifierCase identifierCase) {
+ this.identifierCase = identifierCase;
+ }
+
+ public void setTargetCatalogName(String targetCatalogName) {
+ this.targetCatalogName = targetCatalogName;
+ }
+
+ public void setTargetSchemaName(String targetSchemaName) {
+ this.targetSchemaName = targetSchemaName;
+ }
+
+ public void setTargetVersion(int majorVersion, int minorVersion) {
+ this.targetMajorVersion = majorVersion;
+ this.targetMinorVersion = minorVersion;
+ }
+
+ /**
+ * Compression of Parquet files: ZSTD (default), SNAPPY, GZIP or UNCOMPRESSED
+ */
+ public void setCompression(String compression) {
+ this.compression = StringUtils.defaultIfBlank(compression, DEFAULT_COMPRESSION).toUpperCase();
+ }
+
+ public String getCompression() {
+ return compression;
+ }
+
+ public File getRoot() {
+ return root;
+ }
+
+ public List getWrittenFiles() {
+ return new ArrayList<>(writtenFiles);
+ }
+
+ public void exportDdlAndData() throws Exception {
+ export(ExportMode.DDL_DATA);
+ }
+
+ public void exportDdl() throws Exception {
+ export(ExportMode.DDL);
+ }
+
+ public void exportData() throws Exception {
+ export(ExportMode.DATA);
+ }
+
+ public void export(ExportMode exportMode) throws Exception {
+ writtenFiles.clear();
+ if (configuration.getDbType() == null) {
+ configuration.setDbType(DbType.forUrl(configuration.getUrl()));
+ }
+ boolean successful = false;
+ exportListener.onStart(exportMode);
+ PhaseListener phaseListener = new PhaseListener(exportListener);
+ try {
+ ExportManifest manifest;
+ if (exportMode != ExportMode.DATA) {
+ exportSchema(phaseListener);
+ manifest = ExportManifest.read(root);
+ } else {
+ manifest = null;
+ }
+ if (manifest == null) {
+ manifest = new ExportManifest();
+ manifest.getSource().setDbType(configuration.getDbType());
+ manifest.getTarget().setDbType(targetDbType != null ? targetDbType : configuration.getDbType());
+ }
+ List parquetFiles = new ArrayList<>();
+ if (exportMode != ExportMode.DDL) {
+ exportRows(phaseListener, manifest, parquetFiles);
+ }
+ writeManifest(manifest, exportMode, parquetFiles);
+ successful = !phaseListener.isFailed();
+ } catch (ExportCancelledException e) {
+ throw e;
+ } catch (Exception e) {
+ exportListener.onError(e.getMessage(), e);
+ throw e;
+ } finally {
+ exportListener.onEnd(successful);
+ }
+ }
+
+ /**
+ * schema.sql and constraints.sql by the script exporter
+ */
+ private void exportSchema(PhaseListener listener) throws Exception {
+ exportListener.onMessage("Export schema");
+ ScriptExporter scriptExporter = new ScriptExporter(root, DataFileStrategy.SINGLE_FILE, 0);
+ copyConfiguration(configuration, scriptExporter.getConfiguration());
+ scriptExporter.setTargetDbType(targetDbType);
+ scriptExporter.setIdentifierCase(identifierCase);
+ scriptExporter.setTargetCatalogName(targetCatalogName);
+ scriptExporter.setTargetSchemaName(targetSchemaName);
+ if (targetMajorVersion >= 0) {
+ scriptExporter.setTargetVersion(targetMajorVersion, targetMinorVersion);
+ }
+ scriptExporter.setConstraintsSeparated(true);
+ scriptExporter.setExportListener(listener.forPhase(false));
+ scriptExporter.exportDdl();
+ writtenFiles.addAll(scriptExporter.getWrittenFiles());
+ }
+
+ /**
+ * Rows are copied into the workspace and written as Parquet files, one file per table
+ */
+ private void exportRows(PhaseListener listener, ExportManifest manifest,
+ List parquetFiles) throws Exception {
+ try (DuckDBWorkspace workspace = new DuckDBWorkspace(root)) {
+ exportListener.onMessage("Copy rows into workspace: " + workspace.getFile());
+ ImportExporter importExporter = new ImportExporter();
+ copyConfiguration(configuration, importExporter.getExportConfiguration());
+ Exporter.ExportConfiguration source = importExporter.getExportConfiguration();
+ source.setIdReused(true);
+ source.setTableRecreated(true);
+ source.setIndexIncluded(false);
+ source.setForeignKeyIncluded(false);
+ source.setCommentIncluded(false);
+ // Defaults of the same database type may use sequences
+ source.setSequenceIncluded(true);
+ ImportExportHandler.ImportConfiguration target = importExporter.getImportConfiguration();
+ target.setDbType(DbType.DUCKDB);
+ target.setUrl(workspace.getUrl());
+ target.setConnectionPoolEnabled(false);
+ DuckDBDialect dialect = new DuckDBDialect();
+ dialect.setExactValues(true);
+ dialect.setExactTargetDbType(targetDbType != null ? targetDbType : configuration.getDbType());
+ // Table names of the source are kept
+ dialect.setIdentifierCase(IdentifierCase.KEEP);
+ importExporter.setTargetDialect(dialect);
+ importExporter.setTargetConnectionFactory(new SharedConnectionFactory(workspace.getConnection()));
+ importExporter.setExportListener(listener.forPhase(true));
+ importExporter.exportDdlAndData();
+ if (listener.isFailed()) {
+ return;
+ }
+
+ List tables = workspace.getTables();
+ boolean schemaQualified = tables.stream().map(t -> t[0]).distinct().count() > 1;
+ DuckDBMetaDataOperations operations = new DuckDBMetaDataOperations();
+ DatabaseMetaData metaData = workspace.getConnection().getMetaData();
+ for (String[] table : tables) {
+ if (exportListener.isCancelled()) {
+ throw new ExportCancelledException();
+ }
+ String schema = table[0];
+ String tableName = table[1];
+ String fileName = (schemaQualified ? LobManifestWriter.toFileName(schema) + "." : "")
+ + LobManifestWriter.toFileName(tableName) + PARQUET_EXTENSION;
+ File file = new File(new File(root, ScriptExportHandler.DATA_DIR_NAME), fileName);
+ Map fileMetadata = new LinkedHashMap<>();
+ fileMetadata.put("sqljam_table", tableName);
+ if (sourceSchema(schema) != null) {
+ fileMetadata.put("sqljam_schema", sourceSchema(schema));
+ }
+ fileMetadata.put("sqljam_source", String.valueOf(configuration.getDbType()));
+ workspace.writeParquet(schema, tableName, file, compression, fileMetadata);
+ writtenFiles.add(file);
+ long rows = workspace.countRows(schema, tableName);
+ exportListener.onMessage(String.format("Write %d rows of %s to %s", rows, tableName, file));
+
+ ExportManifest.TableEntry tableEntry = findTableEntry(manifest, sourceSchema(schema), tableName)
+ .orElseGet(() -> {
+ ExportManifest.TableEntry entry = new ExportManifest.TableEntry();
+ entry.setSchema(sourceSchema(schema));
+ entry.setName(tableName);
+ manifest.getTables().add(entry);
+ return entry;
+ });
+ tableEntry.setRows(rows);
+ String path = ScriptExportHandler.DATA_DIR_NAME + "/" + fileName;
+ tableEntry.getDataFiles().clear();
+ tableEntry.getDataFiles().add(path);
+ tableEntry.getIdentityColumns().clear();
+ for (Map column : operations.getColumnInfos(metaData, null, schema, tableName)) {
+ if ("YES".equalsIgnoreCase((String) column.get("IS_AUTOINCREMENT"))) {
+ tableEntry.getIdentityColumns().add((String) column.get("COLUMN_NAME"));
+ }
+ }
+ parquetFiles.add(new ExportManifest.FileEntry(path, ExportManifest.FileType.PARQUET));
+ }
+ }
+ }
+
+ /**
+ * Schema of the source table, tables of sources without schemas are in the default schema of the workspace
+ */
+ private String sourceSchema(String workspaceSchema) {
+ return configuration.getDbType() != null && !configuration.getDbType().isSchemaSupported()
+ && DuckDBWorkspace.DEFAULT_SCHEMA.equals(workspaceSchema) ? null : workspaceSchema;
+ }
+
+ private static Optional findTableEntry(ExportManifest manifest, String schema,
+ String table) {
+ return manifest.getTables().stream()
+ .filter(entry -> table.equals(entry.getName())
+ && (schema == null || entry.getSchema() == null || schema.equals(entry.getSchema())))
+ .findFirst();
+ }
+
+ /**
+ * Files in the order of importing: schema.sql, Parquet files, constraints.sql
+ */
+ private void writeManifest(ExportManifest manifest, ExportMode exportMode,
+ List parquetFiles) throws Exception {
+ manifest.setDataFormat(DataFormat.PARQUET);
+ manifest.setExportMode(exportMode);
+ manifest.setStatus(ExportManifest.Status.COMPLETED);
+ manifest.getOptions().put("compression", compression);
+ List files = new ArrayList<>();
+ for (ExportManifest.FileEntry entry : manifest.getFiles()) {
+ if (entry.getType() == ExportManifest.FileType.SCHEMA) {
+ files.add(entry);
+ }
+ }
+ files.addAll(parquetFiles);
+ for (ExportManifest.FileEntry entry : manifest.getFiles()) {
+ if (entry.getType() == ExportManifest.FileType.CONSTRAINTS) {
+ files.add(entry);
+ }
+ }
+ manifest.setFiles(files);
+ manifest.write(root);
+ File manifestFile = new File(root, ExportManifest.FILE_NAME);
+ if (!writtenFiles.contains(manifestFile)) {
+ writtenFiles.add(manifestFile);
+ }
+ exportListener.onMessage("Write manifest: " + manifestFile);
+ }
+
+ /**
+ * Copies source settings and options, dialects are created by each exporter
+ */
+ static void copyConfiguration(Exporter.ExportConfiguration from, Exporter.ExportConfiguration to) {
+ for (Field field : Exporter.ExportConfiguration.class.getDeclaredFields()) {
+ if (Modifier.isStatic(field.getModifiers()) || "dialect".equals(field.getName())
+ || "sourceDialect".equals(field.getName())) {
+ continue;
+ }
+ try {
+ field.setAccessible(true);
+ field.set(to, field.get(from));
+ } catch (IllegalAccessException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+ }
+
+ /**
+ * Forwards events of the schema phase and the rows phase as one export: messages and errors of both phases,
+ * progress of the rows phase only, start and end are reported by the parquet exporter
+ */
+ static class PhaseListener {
+
+ private final ExportListener delegate;
+ private boolean failed;
+
+ PhaseListener(ExportListener delegate) {
+ this.delegate = delegate;
+ }
+
+ boolean isFailed() {
+ return failed;
+ }
+
+ ExportListener forPhase(boolean progress) {
+ return new ExportListener() {
+
+ @Override
+ public void onTableStart(String catalog, String schema, String table, long totalRows) {
+ if (progress) {
+ delegate.onTableStart(catalog, schema, table, totalRows);
+ }
+ }
+
+ @Override
+ public void onTableProgress(String catalog, String schema, String table, long rows,
+ long totalRows) {
+ if (progress) {
+ delegate.onTableProgress(catalog, schema, table, rows, totalRows);
+ }
+ }
+
+ @Override
+ public void onTableEnd(String catalog, String schema, String table, long rows) {
+ if (progress) {
+ delegate.onTableEnd(catalog, schema, table, rows);
+ }
+ }
+
+ @Override
+ public void onProgress(long processed, long total) {
+ if (progress) {
+ delegate.onProgress(processed, total);
+ }
+ }
+
+ @Override
+ public void onMessage(String message) {
+ delegate.onMessage(message);
+ }
+
+ @Override
+ public void onError(String message, Throwable e) {
+ failed = true;
+ delegate.onError(message, e);
+ }
+
+ @Override
+ public void onEnd(boolean successful) {
+ if (!successful) {
+ failed = true;
+ }
+ }
+
+ @Override
+ public boolean isCancelled() {
+ return delegate.isCancelled();
+ }
+ };
+ }
+ }
+}
diff --git a/src/main/java/com/github/sqljam/impexp/ParquetImporter.java b/src/main/java/com/github/sqljam/impexp/ParquetImporter.java
new file mode 100644
index 0000000..220ab71
--- /dev/null
+++ b/src/main/java/com/github/sqljam/impexp/ParquetImporter.java
@@ -0,0 +1,436 @@
+/*
+ * Copyright 2023-2026 Fred Feng
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.github.sqljam.impexp;
+
+import java.io.File;
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.Validate;
+import com.github.sqljam.jdbc.SharedConnectionFactory;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * @Description: ParquetImporter imports Parquet files into a database by a connection. Packages exported by
+ * {@link ParquetExporter} are imported in the order of manifest.json: schema.sql, Parquet files into
+ * the created tables (identity values are kept and identities are reset), constraints.sql. Parquet
+ * files of other tools are imported into a new table created from the Parquet schema, or appended
+ * to an existing table. Files are read into a temporary DuckDB workspace, then rows are copied by
+ * {@link ImportExporter} with the dialect of the target database.
+ * @Author: Fred Feng
+ * @Date: 26/03/2023
+ * @Version 1.0.0
+ */
+@Slf4j
+public class ParquetImporter extends AbstractFileImporter {
+
+ /**
+ * CREATE drops and creates the table from the Parquet schema, APPEND inserts rows into the existing table
+ */
+ public enum Mode {
+ CREATE,
+ APPEND
+ }
+
+ private String targetCatalogName;
+ private String targetSchemaName;
+ private IdentifierCase identifierCase;
+ private int targetMajorVersion = -1;
+ private int targetMinorVersion = -1;
+ private int pageSize = Exporter.DEFAULT_PAGE_SIZE;
+ private long importedRows;
+
+ public ParquetImporter(Connection connection, DbType dbType) {
+ super(connection, dbType);
+ }
+
+ /**
+ * Catalog of the table, the catalog of the connection by default
+ */
+ public void setTargetCatalogName(String targetCatalogName) {
+ this.targetCatalogName = targetCatalogName;
+ }
+
+ /**
+ * Schema of the table, the schema of the connection by default
+ */
+ public void setTargetSchemaName(String targetSchemaName) {
+ this.targetSchemaName = targetSchemaName;
+ }
+
+ public void setIdentifierCase(IdentifierCase identifierCase) {
+ this.identifierCase = identifierCase;
+ }
+
+ public void setTargetVersion(int majorVersion, int minorVersion) {
+ this.targetMajorVersion = majorVersion;
+ this.targetMinorVersion = minorVersion;
+ }
+
+ public void setPageSize(int pageSize) {
+ this.pageSize = pageSize;
+ }
+
+ /**
+ * Rows imported from Parquet files
+ */
+ public long getImportedRows() {
+ return importedRows;
+ }
+
+ /**
+ * Imports Parquet files of other tools into a table
+ *
+ * @return number of imported rows
+ */
+ public long importFiles(List files, String tableName, Mode mode) throws Exception {
+ Validate.notEmpty(files, "Parquet files must be required");
+ Validate.notBlank(tableName, "Table name must be required");
+ importedRows = 0;
+ boolean successful = false;
+ exportListener.onStart(mode == Mode.CREATE ? ExportMode.DDL_DATA : ExportMode.DATA);
+ try (DuckDBWorkspace workspace = new DuckDBWorkspace(workspaceDirectory)) {
+ long total = workspace.countParquetRows(files);
+ exportListener.onMessage(String.format("Import %d rows of %d Parquet files into %s", total, files.size(),
+ tableName));
+ importTable(workspace, DuckDBWorkspace.DEFAULT_SCHEMA, tableName, files, Collections.emptyList(), mode,
+ null, null, new ProgressListener(exportListener, 0, total));
+ executedCount = (int) Math.min(Integer.MAX_VALUE, importedRows);
+ successful = true;
+ } finally {
+ exportListener.onEnd(successful);
+ }
+ return importedRows;
+ }
+
+ /**
+ * schema.sql, Parquet files (all together, in the order of foreign keys) and constraints.sql
+ */
+ @Override
+ protected void importManifest(File dir, ExportManifest manifest) throws IOException, SQLException {
+ if (manifest.getDataFormat() != DataFormat.PARQUET) {
+ throw new ImpExpException("Packages of sql scripts are imported by ScriptImporter");
+ }
+ importedRows = 0;
+ boolean parquetImported = false;
+ for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) {
+ File file = new File(dir, fileEntry.getPath());
+ if (fileEntry.getType() == ExportManifest.FileType.PARQUET) {
+ if (!parquetImported) {
+ importPackageFiles(dir, manifest);
+ parquetImported = true;
+ }
+ } else {
+ runScript(file);
+ }
+ }
+ }
+
+ private void importPackageFiles(File dir, ExportManifest manifest) throws SQLException {
+ long parquetBytes = manifest.getFiles().stream()
+ .filter(entry -> entry.getType() == ExportManifest.FileType.PARQUET)
+ .mapToLong(entry -> new File(dir, entry.getPath()).length()).sum();
+ long startBytes = completedBytes;
+ ExportListener bytesListener = new ForwardingListener(exportListener) {
+
+ @Override
+ public void onProgress(long processed, long total) {
+ // Rows are converted to bytes of Parquet files
+ long bytes = total > 0 ? parquetBytes * processed / total : 0;
+ exportListener.onProgress(Math.min(startBytes + bytes, totalBytes), totalBytes);
+ }
+
+ @Override
+ public void onError(String message, Throwable e) {
+ failedCount++;
+ exportListener.onError(message, e);
+ }
+ };
+ List tables = manifest.getTables().stream()
+ .filter(table -> !table.getDataFiles().isEmpty()).collect(Collectors.toList());
+ IdentifierCase manifestCase = getIdentifierCase(manifest);
+ if (identifierCase == null) {
+ identifierCase = manifestCase;
+ }
+ try {
+ if (createDialect().isForeignKeyOrderRequired()) {
+ tables = sortByForeignKeys(tables, manifest);
+ }
+ long total = tables.stream().mapToLong(ExportManifest.TableEntry::getRows).sum();
+ long completed = 0;
+ try (DuckDBWorkspace workspace = new DuckDBWorkspace(workspaceDirectory)) {
+ for (ExportManifest.TableEntry table : tables) {
+ if (exportListener.isCancelled()) {
+ throw new ExportCancelledException();
+ }
+ List files = table.getDataFiles().stream().map(path -> new File(dir, path))
+ .collect(Collectors.toList());
+ String schema = StringUtils.defaultIfBlank(table.getSchema(), DuckDBWorkspace.DEFAULT_SCHEMA);
+ exportListener.onMessage(String.format("Import %d rows of %s", table.getRows(), table.getName()));
+ importTable(workspace, schema, table.getName(), files, table.getIdentityColumns(), Mode.APPEND,
+ manifest.getTarget().getCatalog(), manifest.getTarget().getSchema(),
+ new ProgressListener(bytesListener, completed, total));
+ completed += table.getRows();
+ }
+ }
+ } catch (ImpExpException | SQLException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new ImpExpException(e.getMessage(), e);
+ }
+ executedCount += (int) Math.min(Integer.MAX_VALUE, importedRows);
+ completedBytes = startBytes;
+ completeBytes(parquetBytes);
+ }
+
+ private void importTable(DuckDBWorkspace workspace, String schema, String tableName, List files,
+ List identityColumns, Mode mode, String packageCatalog, String packageSchema,
+ ProgressListener listener) throws Exception {
+ workspace.loadParquet(schema, tableName, files, identityColumns);
+ String catalog = StringUtils.defaultIfBlank(targetCatalogName, packageCatalog);
+ String targetSchema = StringUtils.defaultIfBlank(targetSchemaName, packageSchema);
+ if (mode == Mode.APPEND) {
+ // Values of generated and row version columns are maintained by the target database
+ List generated = getGeneratedColumns(tableName, catalog, targetSchema);
+ List dropped = new ArrayList<>();
+ for (String[] column : workspace.describeTable(schema, tableName)) {
+ if (generated.stream().anyMatch(name -> name.equalsIgnoreCase(column[0]))) {
+ dropped.add(column[0]);
+ }
+ }
+ workspace.dropColumns(schema, tableName, dropped);
+ }
+ ImportExporter importExporter = new ImportExporter();
+ Exporter.ExportConfiguration source = importExporter.getExportConfiguration();
+ source.setDbType(DbType.DUCKDB);
+ source.setUrl(workspace.getUrl());
+ source.setConnectionPoolEnabled(false);
+ source.setIncludedSchemaNames(new String[]{schema});
+ source.setIncludedTableNames(new String[]{tableName});
+ source.setPageSize(pageSize);
+ source.setIdReused(true);
+ source.setTableRecreated(mode == Mode.CREATE);
+ source.setIndexIncluded(false);
+ source.setForeignKeyIncluded(false);
+ source.setCommentIncluded(false);
+ source.setSequenceIncluded(false);
+ ImportExportHandler.ImportConfiguration target = importExporter.getImportConfiguration();
+ target.setDbType(dbType);
+ target.setTargetCatalogName(catalog);
+ target.setTargetSchemaName(targetSchema);
+ importExporter.setTargetConnectionFactory(new SharedConnectionFactory(connection));
+ if (identifierCase != null) {
+ importExporter.setIdentifierCase(identifierCase);
+ }
+ if (targetMajorVersion >= 0) {
+ importExporter.setTargetVersion(targetMajorVersion, targetMinorVersion);
+ }
+ // Tables go into the target schema or the default schema, not into a schema named as the workspace one
+ importExporter.setSourceSchemaPreserved(false);
+ importExporter.setExportListener(listener);
+ // The workspace is read by connections of the importer
+ workspace.closeConnection();
+ importExporter.export(mode == Mode.CREATE ? ExportMode.DDL_DATA : ExportMode.DATA);
+ if (listener.getError() != null) {
+ throw new ImpExpException(listener.getError(), listener.getCause());
+ }
+ importedRows += listener.getRows();
+ }
+
+ /**
+ * Dialect folding names as the importer does: the workspace (DuckDB) is the source
+ */
+ private Dialect createDialect() {
+ Dialect dialect = dbType.createDialect();
+ dialect.setSourceDbType(DbType.DUCKDB);
+ if (identifierCase != null) {
+ dialect.setIdentifierCase(identifierCase);
+ }
+ return dialect;
+ }
+
+ private static IdentifierCase getIdentifierCase(ExportManifest manifest) {
+ Object option = manifest.getOptions().get("identifierCase");
+ if (option == null) {
+ return null;
+ }
+ try {
+ return IdentifierCase.valueOf(option.toString());
+ } catch (IllegalArgumentException e) {
+ return null;
+ }
+ }
+
+ /**
+ * Generated (computed) and row version columns of the existing target table
+ */
+ private List getGeneratedColumns(String tableName, String catalog, String schema) throws SQLException {
+ List columns = new ArrayList<>();
+ MetaDataOperations operations = dbType.createMetaDataOperations();
+ String tableSchema = StringUtils.defaultIfBlank(schema, connection.getSchema());
+ String tableCatalog = dbType.isCatalogSupported() && dbType != DbType.ORACLE
+ ? StringUtils.defaultIfBlank(catalog, connection.getCatalog()) : null;
+ List> columnInfos;
+ try {
+ columnInfos = operations.getColumnInfos(connection.getMetaData(), tableCatalog, tableSchema,
+ createDialect().foldIdentifier(tableName));
+ } catch (ImpExpException e) {
+ // The table does not exist, the importer reports it
+ columnInfos = Collections.emptyList();
+ }
+ for (Map column : columnInfos) {
+ if ("YES".equalsIgnoreCase((String) column.get("IS_GENERATEDCOLUMN"))
+ || Boolean.TRUE.equals(column.get("IS_ROWVERSION"))) {
+ columns.add((String) column.get("COLUMN_NAME"));
+ }
+ }
+ return columns;
+ }
+
+ /**
+ * Referenced tables are imported first, foreign keys are read from the tables created by schema.sql
+ */
+ private List sortByForeignKeys(List tables,
+ ExportManifest manifest) throws SQLException {
+ Dialect dialect = createDialect();
+ Map byName = new LinkedHashMap<>();
+ for (ExportManifest.TableEntry table : tables) {
+ byName.put(dialect.foldIdentifier(table.getName()).toLowerCase(), table);
+ }
+ DatabaseMetaData metaData = connection.getMetaData();
+ MetaDataOperations operations = dbType.createMetaDataOperations();
+ String schema = StringUtils.defaultIfBlank(targetSchemaName,
+ StringUtils.defaultIfBlank(manifest.getTarget().getSchema(), connection.getSchema()));
+ Map> dependencies = new LinkedHashMap<>();
+ for (Map.Entry entry : byName.entrySet()) {
+ Set referenced = new LinkedHashSet<>();
+ for (Map keyInfo : operations.getImportedKeyInfos(metaData, null, schema,
+ dialect.foldIdentifier(entry.getValue().getName()))) {
+ String refTable = StringUtils.lowerCase((String) keyInfo.get("PKTABLE_NAME"));
+ if (refTable != null && !refTable.equals(entry.getKey()) && byName.containsKey(refTable)) {
+ referenced.add(refTable);
+ }
+ }
+ dependencies.put(entry.getKey(), referenced);
+ }
+ List sorted = new ArrayList<>();
+ Set visited = new HashSet<>();
+ for (String name : byName.keySet()) {
+ addInOrder(name, byName, dependencies, visited, new HashSet<>(), sorted);
+ }
+ return sorted;
+ }
+
+ private static void addInOrder(String name, Map tables,
+ Map> dependencies, Set visited, Set visiting,
+ List sorted) {
+ if (visited.contains(name) || !visiting.add(name)) {
+ return;
+ }
+ for (String referenced : dependencies.getOrDefault(name, Collections.emptySet())) {
+ addInOrder(referenced, tables, dependencies, visited, visiting, sorted);
+ }
+ visited.add(name);
+ sorted.add(tables.get(name));
+ }
+
+ /**
+ * Forwards events of a table import: progress of the table is added to rows of tables imported before, start
+ * and end of the inner import are not forwarded, the first error is kept
+ */
+ static class ProgressListener implements ExportListener {
+
+ private final ExportListener delegate;
+ private final long completed;
+ private final long total;
+ private long rows;
+ private String error;
+ private Throwable cause;
+
+ ProgressListener(ExportListener delegate, long completed, long total) {
+ this.delegate = delegate;
+ this.completed = completed;
+ this.total = total;
+ }
+
+ long getRows() {
+ return rows;
+ }
+
+ String getError() {
+ return error;
+ }
+
+ Throwable getCause() {
+ return cause;
+ }
+
+ @Override
+ public void onTableStart(String catalogName, String schemaName, String tableName, long totalRows) {
+ delegate.onTableStart(catalogName, schemaName, tableName, totalRows);
+ }
+
+ @Override
+ public void onTableProgress(String catalogName, String schemaName, String tableName, long processedRows,
+ long totalRows) {
+ delegate.onTableProgress(catalogName, schemaName, tableName, processedRows, totalRows);
+ }
+
+ @Override
+ public void onTableEnd(String catalogName, String schemaName, String tableName, long processedRows) {
+ rows += processedRows;
+ delegate.onTableEnd(catalogName, schemaName, tableName, processedRows);
+ }
+
+ @Override
+ public void onProgress(long processed, long tableTotal) {
+ if (total > 0) {
+ delegate.onProgress(Math.min(completed + processed, total), total);
+ }
+ }
+
+ @Override
+ public void onMessage(String message) {
+ delegate.onMessage(message);
+ }
+
+ @Override
+ public void onError(String message, Throwable e) {
+ if (error == null) {
+ error = message;
+ cause = e;
+ }
+ delegate.onError(message, e);
+ }
+
+ @Override
+ public boolean isCancelled() {
+ return delegate.isCancelled();
+ }
+ }
+}
diff --git a/src/main/java/com/github/sqljam/impexp/PartitionMetaData.java b/src/main/java/com/github/sqljam/impexp/PartitionMetaData.java
index e9abcc9..92af399 100644
--- a/src/main/java/com/github/sqljam/impexp/PartitionMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/PartitionMetaData.java
@@ -22,11 +22,10 @@
import java.util.Map;
import java.util.Optional;
+import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.lang3.ArrayUtils;
-
/**
* @Description: PartitionMetaData generates the partition clause of a partitioned table and its partitions (PostgreSQL)
* @Author: Fred Feng
@@ -130,7 +129,7 @@ public List getPartitionTableMetaDatas() {
public String[] getStatements() throws SQLException {
String catalogName = getCatalogName();
String schemaName = getSchemaName();
- String tableName = getTableName();
+ String tableName = getDialect().getTargetTableName(getTableName());
String partitionType = (String) detail.get("PARTITION_TYPE");
String columnNames = (String) detail.get("PARTITION_COLUMN_NAMES");
String statement = getDialect().getDefinePartitionTableStatement(catalogName, schemaName, tableName,
diff --git a/src/main/java/com/github/sqljam/impexp/PartitionTableMetaData.java b/src/main/java/com/github/sqljam/impexp/PartitionTableMetaData.java
index a88c676..acd6676 100644
--- a/src/main/java/com/github/sqljam/impexp/PartitionTableMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/PartitionTableMetaData.java
@@ -20,8 +20,8 @@
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
-import java.util.stream.Collectors;
import java.util.Map;
+import java.util.stream.Collectors;
/**
* @Description: PartitionTableMetaData is a partition of a partitioned table of PostgreSQL
@@ -114,9 +114,10 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
public String[] getStatements() throws SQLException {
String catalogName = getCatalogName();
String schemaName = getSchemaName();
- String tableName = getTableName();
- String inheritedTableName = (String) detail.get("INHERITED_TABLE_NAME");
- String statement = getDialect().getCreatePartitionTableStatement(catalogName, schemaName, tableName,
+ Dialect dialect = getDialect();
+ String tableName = dialect.getTargetTableName(getTableName());
+ String inheritedTableName = dialect.getTargetTableName((String) detail.get("INHERITED_TABLE_NAME"));
+ String statement = dialect.getCreatePartitionTableStatement(catalogName, schemaName, tableName,
inheritedTableName);
return new String[]{statement};
}
diff --git a/src/main/java/com/github/sqljam/impexp/PrimaryKeyMetaData.java b/src/main/java/com/github/sqljam/impexp/PrimaryKeyMetaData.java
index 5cb87fd..c15ea04 100644
--- a/src/main/java/com/github/sqljam/impexp/PrimaryKeyMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/PrimaryKeyMetaData.java
@@ -89,8 +89,9 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
public String[] getStatements() throws SQLException {
String catalogName = getCatalogName();
String schemaName = getSchemaName();
- String tableName = getTableName();
- String pkName = (String) detail.get("PK_NAME");
+ String tableName = getDialect().getTargetTableName(getTableName());
+ // Names of copied keys are derived from the target table
+ String pkName = getDialect().isTableRenamed() ? null : (String) detail.get("PK_NAME");
String statement = getDialect().getCreatePrimaryKeyStatement(catalogName, schemaName, tableName, columnName,
pkName);
return new String[]{statement};
diff --git a/src/main/java/com/github/sqljam/impexp/SchemaMetaData.java b/src/main/java/com/github/sqljam/impexp/SchemaMetaData.java
index 5886891..14074a6 100644
--- a/src/main/java/com/github/sqljam/impexp/SchemaMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/SchemaMetaData.java
@@ -19,9 +19,16 @@
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
+import java.util.regex.Pattern;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
@@ -61,17 +68,24 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
Exporter.ExportConfiguration configuration = visitor.getConfiguration();
List> tableInfos = getMetaDataOperations().getTableInfos(getMetaData(), getCatalogName(),
schemaName);
+ Set includedTableNames = new HashSet<>();
for (Map tableInfo : tableInfos) {
String tableName = (String) tableInfo.get("TABLE_NAME");
- if (StringUtils.isNotBlank(configuration.getIncludedTableNamePattern()) &&
- !tableName.matches(configuration.getIncludedTableNamePattern())) {
- continue;
- }
- if (ArrayUtils.isEmpty(configuration.getIncludedTableNames())
- || ArrayUtils.contains(configuration.getIncludedTableNames(), tableName)) {
+ if (isIncluded(configuration, tableName)) {
+ includedTableNames.add(tableName);
+ }
+ }
+ for (Map tableInfo : tableInfos) {
+ String tableName = (String) tableInfo.get("TABLE_NAME");
+ // Partitions of included partitioned tables are included (PostgreSQL), rows are stored in them
+ if (includedTableNames.contains(tableName) || (Boolean.TRUE.equals(tableInfo.get("IS_PARTITION_TABLE"))
+ && includedTableNames.contains((String) tableInfo.get("INHERITED_TABLE_NAME")))) {
tableMetaDatas.add(new TableMetaData(tableName, tableInfo, this));
}
}
+ if (configuration.getDialect() != null && configuration.getDialect().isForeignKeyOrderRequired()) {
+ sortByForeignKeys();
+ }
for (TableMetaData tableMetaData : tableMetaDatas) {
tableMetaData.accept(visitor);
}
@@ -94,6 +108,15 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
}
}
+ private static boolean isIncluded(Exporter.ExportConfiguration configuration, String tableName) {
+ if (StringUtils.isNotBlank(configuration.getIncludedTableNamePattern())
+ && !tableName.matches(configuration.getIncludedTableNamePattern())) {
+ return false;
+ }
+ return ArrayUtils.isEmpty(configuration.getIncludedTableNames())
+ || ArrayUtils.contains(configuration.getIncludedTableNames(), tableName);
+ }
+
@Override
public String getCatalogName() {
return tiedMetaData.getCatalogName();
@@ -116,6 +139,51 @@ public T unwrap(Class clz) {
/**
* Whether the sequence is used by the exported tables, by column default values or triggers
*/
+ /**
+ * Referenced tables are placed before referencing tables, the original order is kept otherwise. Tables in a
+ * reference cycle keep their original order.
+ */
+ private void sortByForeignKeys() throws SQLException {
+ Map tables = new LinkedHashMap<>();
+ for (TableMetaData tableMetaData : tableMetaDatas) {
+ tables.put(tableMetaData.getTableName(), tableMetaData);
+ }
+ Map> dependencies = new HashMap<>();
+ for (String tableName : tables.keySet()) {
+ Set referenced = new LinkedHashSet<>();
+ for (Map keyInfo : getMetaDataOperations().getImportedKeyInfos(getMetaData(),
+ getCatalogName(), schemaName, tableName)) {
+ String refTableName = (String) keyInfo.get("PKTABLE_NAME");
+ if (refTableName != null && !refTableName.equals(tableName) && tables.containsKey(refTableName)) {
+ referenced.add(refTableName);
+ }
+ }
+ dependencies.put(tableName, referenced);
+ }
+ List sorted = new ArrayList<>();
+ Set visited = new HashSet<>();
+ Set visiting = new HashSet<>();
+ for (String tableName : tables.keySet()) {
+ addInOrder(tableName, tables, dependencies, visited, visiting, sorted);
+ }
+ tableMetaDatas.clear();
+ tableMetaDatas.addAll(sorted);
+ }
+
+ private static void addInOrder(String tableName, Map tables,
+ Map> dependencies, Set visited, Set visiting,
+ List sorted) {
+ if (visited.contains(tableName) || !visiting.add(tableName)) {
+ return;
+ }
+ for (String refTableName : dependencies.getOrDefault(tableName, Collections.emptySet())) {
+ addInOrder(refTableName, tables, dependencies, visited, visiting, sorted);
+ }
+ visiting.remove(tableName);
+ visited.add(tableName);
+ sorted.add(tables.get(tableName));
+ }
+
private boolean isSequenceReferenced(String sequenceName, Map sequenceInfo) {
String[] referencedTableNames = (String[]) sequenceInfo.get("REFERENCED_TABLE_NAMES");
if (ArrayUtils.isNotEmpty(referencedTableNames)) {
@@ -128,7 +196,7 @@ private boolean isSequenceReferenced(String sequenceName, Map se
String lowerSequenceName = sequenceName.toLowerCase();
return tableMetaDatas.stream().flatMap(md -> md.getColumnMetaDatas().stream())
.map(md -> StringUtils.defaultString((String) md.getDetail().get("COLUMN_DEF")).toLowerCase())
- .anyMatch(def -> def.matches("(?s).*\\b" + java.util.regex.Pattern.quote(lowerSequenceName)
+ .anyMatch(def -> def.matches("(?s).*\\b" + Pattern.quote(lowerSequenceName)
+ "\\b.*"));
}
diff --git a/src/main/java/com/github/sqljam/impexp/ScriptExportHandler.java b/src/main/java/com/github/sqljam/impexp/ScriptExportHandler.java
index 10f72d4..ef92465 100644
--- a/src/main/java/com/github/sqljam/impexp/ScriptExportHandler.java
+++ b/src/main/java/com/github/sqljam/impexp/ScriptExportHandler.java
@@ -19,11 +19,14 @@
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
+import java.sql.DatabaseMetaData;
+import java.sql.Types;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@@ -34,10 +37,10 @@
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
-import com.github.sqljam.jdbc.ConnectionFactory;
-import com.github.sqljam.page.EachPage;
import com.github.sqljam.impexp.DdlScripter.Catalog;
import com.github.sqljam.impexp.DdlScripter.Schema;
+import com.github.sqljam.jdbc.ConnectionFactory;
+import com.github.sqljam.page.EachPage;
import lombok.extern.slf4j.Slf4j;
/**
@@ -125,6 +128,15 @@ File nextFile() {
*/
private List tablePrologue = new ArrayList<>();
+ /**
+ * Foreign keys are always written into constraints.sql, even without rows (Parquet packages load rows later)
+ */
+ private boolean constraintsSeparated;
+
+ public void setConstraintsSeparated(boolean constraintsSeparated) {
+ this.constraintsSeparated = constraintsSeparated;
+ }
+
public void setLobSeparated(boolean lobSeparated) {
this.lobSeparated = lobSeparated;
}
@@ -198,7 +210,7 @@ public void finish(boolean successful) throws Exception {
manifest.getOptions().put("lobSeparated", lobSeparated);
// Files in the order of importing
- Set dirs = new java.util.LinkedHashSet<>();
+ Set dirs = new LinkedHashSet<>();
writtenFiles.forEach(file -> dirs.add(file.getParentFile().getName().equals(DATA_DIR_NAME)
? file.getParentFile().getParentFile() : file.getParentFile()));
for (File dir : dirs) {
@@ -255,7 +267,7 @@ public void prepare(ServerMetaData serverMetaData) throws Exception {
this.dialect = serverMetaData.getDialect();
manifest = new ExportManifest();
manifest.setExportMode(exportMode);
- java.sql.DatabaseMetaData databaseMetaData = serverMetaData.getMetaData();
+ DatabaseMetaData databaseMetaData = serverMetaData.getMetaData();
ExportManifest.Database source = manifest.getSource();
source.setDbType(configuration != null ? configuration.getDbType() : null);
source.setProduct(databaseMetaData.getDatabaseProductName() + " "
@@ -311,7 +323,7 @@ private LobManifestWriter getLobManifestWriter(String catalogName) throws IOExce
return writer;
}
- private static final java.util.Set LOB_TYPE_NAMES = new java.util.HashSet<>(Arrays.asList("text",
+ private static final Set LOB_TYPE_NAMES = new HashSet<>(Arrays.asList("text",
"tinytext", "mediumtext", "longtext", "ntext", "clob", "nclob", "blob", "tinyblob", "mediumblob",
"longblob", "bytea", "image", "long", "long raw", "character large object", "binary large object",
"national character large object"));
@@ -338,9 +350,9 @@ static boolean isLobColumn(ColumnMetaData columnMetaData) {
case "nvarchar":
case "varbinary":
// varchar without length (PostgreSQL), varchar(max) of SQL Server
- return large && (dataType == java.sql.Types.VARCHAR || dataType == java.sql.Types.NVARCHAR
- || dataType == java.sql.Types.VARBINARY || dataType == java.sql.Types.LONGVARCHAR
- || dataType == java.sql.Types.LONGNVARCHAR || dataType == java.sql.Types.LONGVARBINARY);
+ return large && (dataType == Types.VARCHAR || dataType == Types.NVARCHAR
+ || dataType == Types.VARBINARY || dataType == Types.LONGVARCHAR
+ || dataType == Types.LONGNVARCHAR || dataType == Types.LONGVARBINARY);
default:
return false;
}
@@ -414,7 +426,7 @@ public void exportConstraints(DdlScripter ddlScripter) throws Exception {
sqlLines.add(1, String.format("%s Create foreign keys", commentPrefix));
File catalogDir = getCatalogDir(catalogName);
File outputFile;
- if (!dataCatalogNames.contains(StringUtils.defaultString(catalogName))) {
+ if (!constraintsSeparated && !dataCatalogNames.contains(StringUtils.defaultString(catalogName))) {
outputFile = new File(catalogDir, SCHEMA_FILE_NAME);
} else {
// Foreign keys are created after rows inserted and LOBs restored
@@ -472,7 +484,8 @@ public void exportData(String catalogName,
List keyColumnNames = tableMetaData.getPrimaryKeyColumnNames();
Set lobColumnNames = new HashSet<>();
// Rows are located by primary keys when restoring LOBs
- if (lobSeparated && !keyColumnNames.isEmpty() && template.keySet().containsAll(keyColumnNames)) {
+ if (lobSeparated && dialect.isLobSeparationSupported() && !keyColumnNames.isEmpty()
+ && template.keySet().containsAll(keyColumnNames)) {
tableMetaData.getColumnMetaDatas().stream().filter(ScriptExportHandler::isLobColumn)
.map(ColumnMetaData::getColumnName).filter(template::containsKey).forEach(lobColumnNames::add);
}
diff --git a/src/main/java/com/github/sqljam/impexp/ScriptExporter.java b/src/main/java/com/github/sqljam/impexp/ScriptExporter.java
index 7bdd994..5688d14 100644
--- a/src/main/java/com/github/sqljam/impexp/ScriptExporter.java
+++ b/src/main/java/com/github/sqljam/impexp/ScriptExporter.java
@@ -104,6 +104,13 @@ public void setLobSeparated(boolean lobSeparated) {
scriptExportHandler.setLobSeparated(lobSeparated);
}
+ /**
+ * Writes foreign keys into constraints.sql even if rows are not exported (default: into schema.sql)
+ */
+ public void setConstraintsSeparated(boolean constraintsSeparated) {
+ scriptExportHandler.setConstraintsSeparated(constraintsSeparated);
+ }
+
public List getWrittenFiles() {
return scriptExportHandler.getWrittenFiles();
}
diff --git a/src/main/java/com/github/sqljam/impexp/ScriptImporter.java b/src/main/java/com/github/sqljam/impexp/ScriptImporter.java
index 856e02b..25010e0 100644
--- a/src/main/java/com/github/sqljam/impexp/ScriptImporter.java
+++ b/src/main/java/com/github/sqljam/impexp/ScriptImporter.java
@@ -23,6 +23,7 @@
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
+import java.sql.Types;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Comparator;
@@ -38,6 +39,7 @@
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.sqljam.jdbc.JdbcUtils;
import lombok.extern.slf4j.Slf4j;
/**
@@ -49,71 +51,31 @@
* @Version 1.0.0
*/
@Slf4j
-public class ScriptImporter {
+public class ScriptImporter extends AbstractFileImporter {
private static final Pattern PART_FILE = Pattern.compile("^(.*?)(?:_(\\d+))?\\.sql$");
- private final Connection connection;
- private final DbType dbType;
- private boolean stopOnError = true;
- private int batchSize = SqlScriptRunner.DEFAULT_BATCH_SIZE;
- private ExportListener exportListener = ExportListener.NONE;
- private int executedCount;
- private int failedCount;
private int lobCount;
- /**
- * Overall progress in bytes of sql files and LOB files
- */
- private long totalBytes;
- private long completedBytes;
public ScriptImporter(Connection connection, DbType dbType) {
- this.connection = connection;
- this.dbType = dbType;
- }
-
- public void setStopOnError(boolean stopOnError) {
- this.stopOnError = stopOnError;
- }
-
- /**
- * INSERT statements per batch, 1 or less executes statement by statement
- */
- public void setBatchSize(int batchSize) {
- this.batchSize = batchSize;
- }
-
- public void setExportListener(ExportListener exportListener) {
- this.exportListener = exportListener != null ? exportListener : ExportListener.NONE;
- }
-
- public int getExecutedCount() {
- return executedCount;
- }
-
- public int getFailedCount() {
- return failedCount;
+ super(connection, dbType);
}
+ @Override
public int getLobCount() {
return lobCount;
}
/**
- * Imports an export directory. Files are executed in the order of manifest.json if it exists, otherwise they are
- * discovered by the directory layout.
+ * Directories without manifest.json are discovered by the layout: schema.sql, data files, lob/ and
+ * constraints.sql of the directory or its sub directories (catalogs)
*/
- public void importDirectory(File dir) throws IOException, SQLException {
- ExportManifest manifest = ExportManifest.read(dir);
- if (manifest != null) {
- boolean successful = false;
- try {
- importManifest(dir, manifest);
- successful = true;
- } finally {
- exportListener.onEnd(successful);
- }
- return;
+ @Override
+ protected void importDirectoryWithoutManifest(File dir) throws IOException, SQLException {
+ File[] parquetFiles = new File(dir, "data").listFiles((parent, name) -> name.endsWith(
+ ParquetExporter.PARQUET_EXTENSION));
+ if (parquetFiles != null && parquetFiles.length > 0) {
+ throw new ImpExpException("Parquet files without manifest are imported by ParquetImporter: " + dir);
}
List units = new ArrayList<>();
if (isScriptDirectory(dir)) {
@@ -147,62 +109,24 @@ public void importDirectory(File dir) throws IOException, SQLException {
}
/**
- * Validates the manifest and executes its files in order
+ * LOB files are counted by the size of their directory
*/
- private static long sizeOf(File file) {
- if (file.isDirectory()) {
- File[] files = file.listFiles();
- long size = 0;
- if (files != null) {
- for (File child : files) {
- size += sizeOf(child);
- }
- }
- return size;
- }
- return file.exists() ? file.length() : 0;
- }
-
- private void startProgress(long totalBytes) {
- this.totalBytes = Math.max(totalBytes, 1);
- this.completedBytes = 0;
- exportListener.onProgress(0, this.totalBytes);
- }
-
- private void completeBytes(long bytes) {
- completedBytes += bytes;
- exportListener.onProgress(Math.min(completedBytes, totalBytes), totalBytes);
- }
-
- private void importManifest(File dir, ExportManifest manifest) throws IOException, SQLException {
- if (!ExportManifest.FORMAT.equals(manifest.getFormat())) {
- throw new ImpExpException("Unknown export format: " + manifest.getFormat());
- }
- if (manifest.getStatus() != ExportManifest.Status.COMPLETED) {
- throw new ImpExpException("The export is not completed: " + manifest.getStatus());
- }
- DbType targetDbType = manifest.getTarget().getDbType();
- if (dbType != null && targetDbType != null && dbType != targetDbType) {
- throw new ImpExpException(String.format("The scripts are generated for %s, but the target database is %s",
- targetDbType.getDisplayName(), dbType.getDisplayName()));
- }
- for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) {
- File file = new File(dir, fileEntry.getPath());
- if (!file.exists()) {
- throw new ImpExpException("File not found: " + fileEntry.getPath());
- }
- if (StringUtils.isNotBlank(fileEntry.getSha256()) && !fileEntry.getSha256().equals(
- ExportManifest.sha256(file))) {
- throw new ImpExpException("File is modified or corrupted (checksum mismatch): " + fileEntry.getPath());
- }
- }
+ @Override
+ protected long getManifestBytes(File dir, ExportManifest manifest) {
long bytes = 0;
for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) {
File file = new File(dir, fileEntry.getPath());
bytes += fileEntry.getType() == ExportManifest.FileType.LOB_MANIFEST
? sizeOf(new File(file.getParentFile(), LobManifestWriter.LOB_DIR_NAME)) : file.length();
}
- startProgress(bytes);
+ return bytes;
+ }
+
+ @Override
+ protected void importManifest(File dir, ExportManifest manifest) throws IOException, SQLException {
+ if (manifest.getDataFormat() == DataFormat.PARQUET) {
+ throw new ImpExpException("Packages of Parquet files are imported by ParquetImporter");
+ }
for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) {
File file = new File(dir, fileEntry.getPath());
if (fileEntry.getType() == ExportManifest.FileType.LOB_MANIFEST) {
@@ -262,40 +186,13 @@ private static String[] partOf(File file) {
return new String[]{file.getName(), ""};
}
- private void runScript(File file) throws IOException, SQLException {
- if (!file.exists()) {
- return;
- }
- exportListener.onMessage("Executing script: " + file.getName());
- SqlScriptRunner runner = new SqlScriptRunner(connection);
- runner.setStopOnError(stopOnError);
- runner.setBatchSize(batchSize);
- long fileBytes = file.length();
- runner.setExportListener(new ForwardingListener(exportListener) {
-
- @Override
- public void onProgress(long processed, long total) {
- // Progress of statements in the file is converted to bytes
- long bytes = total > 0 ? fileBytes * processed / total : 0;
- exportListener.onProgress(Math.min(completedBytes + bytes, totalBytes), totalBytes);
- }
- });
- try {
- runner.runScript(file);
- } finally {
- executedCount += runner.getExecutedCount();
- failedCount += runner.getFailedCount();
- completeBytes(fileBytes);
- }
- }
-
/**
* Restores LOB values by primary keys: UPDATE table SET column = ? WHERE key = ?
*/
private void restoreLobs(File dir, File manifest) throws IOException, SQLException {
ObjectMapper objectMapper = new ObjectMapper();
boolean autoCommit = connection.getAutoCommit();
- connection.setAutoCommit(false);
+ boolean transactional = JdbcUtils.beginTransaction(connection);
try (JsonParser parser = objectMapper.getFactory().createParser(manifest)) {
Dialect dialect = createDialect(objectMapper, manifest);
while (parser.nextToken() != null) {
@@ -306,12 +203,18 @@ private void restoreLobs(File dir, File manifest) throws IOException, SQLExcepti
}
}
}
- connection.commit();
+ if (transactional) {
+ connection.commit();
+ }
} catch (SQLException | IOException | RuntimeException e) {
- connection.rollback();
+ if (transactional) {
+ connection.rollback();
+ }
throw e;
} finally {
- connection.setAutoCommit(autoCommit);
+ if (transactional) {
+ connection.setAutoCommit(autoCommit);
+ }
}
exportListener.onMessage(String.format("%d LOB values restored", lobCount));
}
@@ -397,7 +300,7 @@ private static void bindKey(PreparedStatement ps, JsonNode key, List key
for (String keyName : keyNames) {
JsonNode value = key.get(keyName);
if (value == null || value.isNull()) {
- ps.setNull(index++, java.sql.Types.VARCHAR);
+ ps.setNull(index++, Types.VARCHAR);
} else if (value.isIntegralNumber()) {
ps.setLong(index++, value.asLong());
} else if (value.isNumber()) {
@@ -409,32 +312,4 @@ private static void bindKey(PreparedStatement ps, JsonNode key, List key
}
}
}
-
- /**
- * Forwards errors of scripts, the end of each script is not the end of the import
- */
- private static class ForwardingListener implements ExportListener {
-
-
- private final ExportListener delegate;
-
- ForwardingListener(ExportListener delegate) {
- this.delegate = delegate;
- }
-
- @Override
- public void onMessage(String message) {
- delegate.onMessage(message);
- }
-
- @Override
- public void onError(String message, Throwable e) {
- delegate.onError(message, e);
- }
-
- @Override
- public boolean isCancelled() {
- return delegate.isCancelled();
- }
- }
}
diff --git a/src/main/java/com/github/sqljam/impexp/SequenceMetaData.java b/src/main/java/com/github/sqljam/impexp/SequenceMetaData.java
index a849267..63b2de5 100644
--- a/src/main/java/com/github/sqljam/impexp/SequenceMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/SequenceMetaData.java
@@ -56,6 +56,10 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
@Override
public String[] getStatements() throws SQLException {
Dialect dialect = getDialect();
+ if (dialect.isTableRenamed()) {
+ // Copies of tables use the sequences of the source tables
+ return null;
+ }
String catalogName = getCatalogName();
String schemaName = getSchemaName();
long startValue = getLong("START_VALUE", 1L);
diff --git a/src/main/java/com/github/sqljam/impexp/SqlScriptRunner.java b/src/main/java/com/github/sqljam/impexp/SqlScriptRunner.java
index 60de517..f86663f 100644
--- a/src/main/java/com/github/sqljam/impexp/SqlScriptRunner.java
+++ b/src/main/java/com/github/sqljam/impexp/SqlScriptRunner.java
@@ -32,6 +32,7 @@
import java.util.regex.Pattern;
import org.apache.commons.lang3.StringUtils;
+import com.github.sqljam.jdbc.JdbcUtils;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
@@ -75,6 +76,11 @@ public class SqlScriptRunner {
private int batchSize = DEFAULT_BATCH_SIZE;
private Boolean oracle;
private ExportListener exportListener = ExportListener.NONE;
+ private Statement statement;
+ /**
+ * Whether statements run in a transaction, false if the database has no transactions (ClickHouse)
+ */
+ private boolean transactional;
@Getter
private int executedCount;
@@ -116,8 +122,9 @@ public void runScript(String script) throws IOException, SQLException {
public void runScript(Reader reader) throws IOException, SQLException {
boolean autoCommit = connection.getAutoCommit();
- connection.setAutoCommit(false);
- try (Statement statement = connection.createStatement()) {
+ transactional = JdbcUtils.beginTransaction(connection);
+ statement = connection.createStatement();
+ try {
List statements = parseStatements(reader);
int total = statements.size();
List batch = new ArrayList<>();
@@ -132,41 +139,78 @@ public void runScript(Reader reader) throws IOException, SQLException {
if (batchSize > 1 && isBatchable(sql)) {
if (batch.isEmpty() && uncommitted > 0) {
// A batch is rolled back alone when it fails
- connection.commit();
+ commit();
uncommitted = 0;
}
batch.add(sql);
if (batch.size() >= batchSize) {
- executeBatch(statement, batch);
+ executeBatch(batch);
exportListener.onProgress(index, total);
}
continue;
}
if (!batch.isEmpty()) {
- executeBatch(statement, batch);
+ executeBatch(batch);
}
- if (execute(statement, sql)) {
+ if (execute(sql)) {
exportListener.onProgress(index, total);
if (++uncommitted >= batchCommitSize) {
- connection.commit();
+ commit();
uncommitted = 0;
exportListener.onMessage(String.format("%d statements executed", executedCount));
}
}
}
if (!batch.isEmpty()) {
- executeBatch(statement, batch);
+ executeBatch(batch);
}
- connection.commit();
+ commit();
exportListener.onProgress(total, total);
} finally {
+ closeStatement();
// A closed connection would hide the original exception
- if (!connection.isClosed()) {
+ if (transactional && !connection.isClosed()) {
connection.setAutoCommit(autoCommit);
}
}
}
+ /**
+ * Commits executed statements, they are committed already if the database has no transactions
+ */
+ private void commit() throws SQLException {
+ if (transactional) {
+ connection.commit();
+ }
+ }
+
+ private void rollback() throws SQLException {
+ if (transactional) {
+ connection.rollback();
+ }
+ }
+
+ /**
+ * The statement in use. DuckDB closes a statement when its sql fails, a new one is created then.
+ */
+ private Statement statement() throws SQLException {
+ if (statement == null || statement.isClosed()) {
+ statement = connection.createStatement();
+ }
+ return statement;
+ }
+
+ private void closeStatement() {
+ if (statement != null) {
+ try {
+ statement.close();
+ } catch (SQLException e) {
+ // Closed already
+ }
+ statement = null;
+ }
+ }
+
static boolean isBatchable(String sql) {
return INSERT_STATEMENT.matcher(sql).matches();
}
@@ -174,33 +218,33 @@ static boolean isBatchable(String sql) {
/**
* Executes and commits a batch. A failed batch is rolled back and executed statement by statement
*/
- private void executeBatch(Statement statement, List batch) throws SQLException {
+ private void executeBatch(List batch) throws SQLException {
try {
if (isOracle()) {
for (String sql : getOracleBlockStatements(batch)) {
- statement.execute(sql);
+ statement().execute(sql);
}
} else {
for (String sql : batch) {
- statement.addBatch(sql);
+ statement().addBatch(sql);
}
- statement.executeBatch();
+ statement().executeBatch();
}
- connection.commit();
+ commit();
executedCount += batch.size();
} catch (SQLException e) {
if (connection.isClosed()) {
throw e;
}
- statement.clearBatch();
- connection.rollback();
+ statement().clearBatch();
+ rollback();
if (log.isDebugEnabled()) {
log.debug("Batch failed, statements are executed one by one: {}", e.getMessage());
}
for (String sql : batch) {
- execute(statement, sql);
+ execute(sql);
}
- connection.commit();
+ commit();
} finally {
batch.clear();
}
@@ -210,10 +254,10 @@ private void executeBatch(Statement statement, List batch) throws SQLExc
* Executes a statement, returns false if it fails and errors are ignored. When errors are ignored, a savepoint
* keeps the transaction usable after a failure (PostgreSQL aborts the whole transaction otherwise)
*/
- private boolean execute(Statement statement, String sql) throws SQLException {
- Savepoint savepoint = stopOnError ? null : connection.setSavepoint();
+ private boolean execute(String sql) throws SQLException {
+ Savepoint savepoint = stopOnError || !transactional ? null : connection.setSavepoint();
try {
- statement.execute(sql);
+ statement().execute(sql);
executedCount++;
releaseSavepoint(savepoint);
return true;
@@ -226,10 +270,12 @@ private boolean execute(Statement statement, String sql) throws SQLException {
}
exportListener.onError(message, e);
if (stopOnError) {
- connection.rollback();
+ rollback();
throw e;
}
- connection.rollback(savepoint);
+ if (savepoint != null) {
+ connection.rollback(savepoint);
+ }
return false;
}
}
diff --git a/src/main/java/com/github/sqljam/impexp/StringHelper.java b/src/main/java/com/github/sqljam/impexp/StringHelper.java
index d8ab47c..7d332ef 100644
--- a/src/main/java/com/github/sqljam/impexp/StringHelper.java
+++ b/src/main/java/com/github/sqljam/impexp/StringHelper.java
@@ -15,8 +15,8 @@
*/
package com.github.sqljam.impexp;
-import lombok.experimental.UtilityClass;
import org.apache.commons.lang3.StringUtils;
+import lombok.experimental.UtilityClass;
/**
* @Description: StringHelper pads and repeats strings for pretty scripts
diff --git a/src/main/java/com/github/sqljam/impexp/TableMetaData.java b/src/main/java/com/github/sqljam/impexp/TableMetaData.java
index 4c32d8a..5cbce1e 100644
--- a/src/main/java/com/github/sqljam/impexp/TableMetaData.java
+++ b/src/main/java/com/github/sqljam/impexp/TableMetaData.java
@@ -23,6 +23,7 @@
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.function.Predicate;
import java.util.stream.Collectors;
import org.apache.commons.collections4.CollectionUtils;
@@ -69,6 +70,10 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
log.info("Begin to process table: {}", tableName);
}
Exporter.ExportConfiguration configuration = visitor.getConfiguration();
+ TableQuery query = configuration != null ? configuration.getTableQuery(tableName) : null;
+ // Selected columns of a query, all columns if none
+ Predicate selected = columnName -> query == null || query.getColumns().isEmpty()
+ || query.getColumns().contains(columnName);
final boolean partitioned = isPartitioned();
DatabaseMetaData databaseMetaData = getMetaData();
List> infoList = null;
@@ -80,10 +85,15 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
List> pkInfos = infoList.stream()
.sorted(Comparator.comparingInt(info -> getInt(info, "KEY_SEQ")))
.collect(Collectors.toList());
- pkInfos.forEach(info -> primaryKeyColumnNames.add((String) info.get("COLUMN_NAME")));
- // Composite primary key is one constraint
- String columnNames = StringUtils.join(primaryKeyColumnNames, ",");
- primaryKeyMetaDatas.add(new PrimaryKeyMetaData(columnNames, pkInfos.get(0), this));
+ List keyColumnNames = pkInfos.stream().map(info -> (String) info.get("COLUMN_NAME"))
+ .collect(Collectors.toList());
+ // A primary key is kept if all of its columns are selected
+ if (keyColumnNames.stream().allMatch(selected)) {
+ primaryKeyColumnNames.addAll(keyColumnNames);
+ // Composite primary key is one constraint
+ String columnNames = StringUtils.join(primaryKeyColumnNames, ",");
+ primaryKeyMetaDatas.add(new PrimaryKeyMetaData(columnNames, pkInfos.get(0), this));
+ }
}
}
@@ -91,7 +101,9 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
getCatalogName(), getSchemaName(), tableName);
for (Map columnInfo : infoList) {
String columnName = (String) columnInfo.get("COLUMN_NAME");
- columnMetaDatas.add(new ColumnMetaData(columnName, columnInfo, this));
+ if (selected.test(columnName)) {
+ columnMetaDatas.add(new ColumnMetaData(columnName, columnInfo, this));
+ }
}
for (ColumnMetaData columnMetaData : columnMetaDatas) {
columnMetaData.accept(visitor);
@@ -112,6 +124,10 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
List subList = MapUtils.getOrCreate(indexMetaDatas, indexName, ArrayList::new);
subList.add(new IndexMetaData(indexName, columnName, indexInfo, this));
}
+ // Indexes are kept if all of their columns are selected
+ indexMetaDatas.values().removeIf(indexes -> !indexes.stream().map(IndexMetaData::getColumnName)
+ .allMatch(selected));
+ removeDuplicateIndexes();
for (Map.Entry> entry : indexMetaDatas.entrySet()) {
if (entry.getValue().size() == 1) {
entry.getValue().get(0).accept(visitor);
@@ -137,6 +153,9 @@ public void accept(MetaDataVisitor visitor) throws SQLException {
}
MapUtils.getOrCreate(fkGroups, fkName, ArrayList::new).add(fkInfo);
}
+ // Foreign keys are kept if all of their columns are selected
+ fkGroups.values().removeIf(fkInfos -> !fkInfos.stream().map(info -> (String) info.get("FKCOLUMN_NAME"))
+ .allMatch(selected));
for (Map.Entry>> entry : fkGroups.entrySet()) {
List> fkInfos = entry.getValue().stream()
.sorted(Comparator.comparingInt(info -> getInt(info, "KEY_SEQ")))
@@ -220,7 +239,7 @@ public MetaDataOperations getMetaDataOperations() {
public String[] getStatements() throws SQLException {
String catalogName = getCatalogName();
String schemaName = getSchemaName();
- String tableName = getTableName();
+ String tableName = getDialect().getTargetTableName(getTableName());
String statement = getDialect().getCreateTableStatement(catalogName, schemaName, tableName);
return new String[]{statement};
}
@@ -241,6 +260,35 @@ public List getPrimaryKeyColumnNames() {
return primaryKeyColumnNames;
}
+ /**
+ * Indexes of the same columns, e.g. an index of H2 for a foreign key next to an index created by the user, would
+ * get the same name in the target database, which also refuses a second index of the same columns. One index of
+ * the columns is kept, a unique index before others.
+ */
+ private void removeDuplicateIndexes() {
+ Map, String> indexNames = new LinkedHashMap<>();
+ for (Map.Entry> entry : new ArrayList<>(indexMetaDatas.entrySet())) {
+ if (entry.getValue().isEmpty()) {
+ continue;
+ }
+ List
@@ -304,7 +311,7 @@ 
- 
- 
+ 