基于doris manager部署Doris存算集群及测试

1. 集群架构

2. 节点规划

3. Doris manager部署

部署地址：部署 Manager | SelectDB Docshttps://docs.selectdb.com/enterprise/management-guide/doris-manager-management/deploy-doris-manager

修改配置文件webserver/conf/manager.conf文件可以配置 WebServer 服务

登录服务器2:8004端口

4. 存算分离集群部署

4.1 目录规划

4.2 详细部署步骤

见官网：部署存算分离集群 | SelectDB Docshttps://docs.selectdb.com/enterprise/management-guide/doris-cluster-management/deploy-cluster/deploy-separating-storage-compute-cluster

大功告成

5. 测试

5.1 Storage Vault

Storage Vault 是 Doris 在存算分离模式中所使用的远程共享存储，可配置一个或多个 Storage Vault，可将不同表存储在不同 Storage Vault 上。

5.1.1 创建Storage Vault

创建基于 HDFS 的存算分离模式 Doris 集群，需要确保所有的节点 (包括 FE / BE 节点、Meta Service) 均有权限访问所指定的 HDFS。

CREATE STORAGE VAULT IF NOT EXISTS hdfs_demo_a PROPERTIES ( "type" = "hdfs", -- required "fs.defaultFS" = "hdfs://hdfs.xxxx.net:9000", -- required "path_prefix" = "bigdata/data_a", -- optional, 一般按照业务名称填写 "hadoop.username" = "doris" ); CREATE STORAGE VAULT IF NOT EXISTS hdfs_demo_b PROPERTIES ( "type" = "hdfs", -- required "fs.defaultFS" = "hdfs://hdfs.xxxx.net:9000", -- required "path_prefix" = "bigdata/data_b", -- optional, 一般按照业务名称填写 "hadoop.username" = "doris" ); CREATE STORAGE VAULT IF NOT EXISTS hdfs_vault PROPERTIES ( "type" = "hdfs", -- required "fs.defaultFS" = "hdfs://hdfs.xxxx.net:9000", -- required "path_prefix" = "doris", -- optional, 一般按照业务名称填写 "hadoop.username" = "doris" );

5.2.2 设置默认Storage Vault

SET hdfs_vault AS DEFAULT STORAGE VAULT

5.2.3 查看Storage Vault

show STORAGE vaults

我们创建3个，doris-manager帮我们部署doris时，自动帮我们创建了一个默认的，这里我新建了一个修改为默认的。

5.2.4 建表测试

5.2.4.1 库没设置，表没设置时，且用户有hdfs_vault\hdfs_demo_a\hdfs_demo_b的权限时

create database if not EXISTS ods; CREATE TABLE IF NOT EXISTS example_tbl_unique ( user_id LARGEINT NOT NULL, user_name VARCHAR(50) NOT NULL, city VARCHAR(20), age SMALLINT, sex TINYINT ) UNIQUE KEY(user_id, user_name) DISTRIBUTED BY HASH(user_id) BUCKETS 10 PROPERTIES ( "enable_unique_key_merge_on_write" = "true" ); show create TABLE ods.example_tbl_unique --结果 CREATE TABLE `example_tbl_unique` ( `user_id` largeint NOT NULL, `user_name` varchar(50) NOT NULL, `city` varchar(20) NULL, `age` smallint NULL, `sex` tinyint NULL ) ENGINE=OLAP UNIQUE KEY(`user_id`, `user_name`) DISTRIBUTED BY HASH(`user_id`) BUCKETS 10 PROPERTIES ( "file_cache_ttl_seconds" = "0", "is_being_synced" = "false", "storage_medium" = "hdd", "storage_format" = "V2", "inverted_index_storage_format" = "V2", "enable_unique_key_merge_on_write" = "true", "light_schema_change" = "true", "storage_vault_id" = "2", "storage_vault_name" = "hdfs_vault", "disable_auto_compaction" = "false", "enable_single_replica_compaction" = "false", "group_commit_interval_ms" = "10000", "group_commit_data_bytes" = "134217728", "enable_mow_light_delete" = "false" );

5.2.4.2 库设置，表没设置时，且用户有hdfs_vault\hdfs_demo_a\hdfs_demo_b的权限时

CREATE DATABASE IF NOT EXISTS `ods_a` PROPERTIES ( "storage_vault_name" = "hdfs_demo_a" ); CREATE TABLE IF NOT EXISTS ods_a.example_tbl_unique ( user_id LARGEINT NOT NULL, user_name VARCHAR(50) NOT NULL, city VARCHAR(20), age SMALLINT, sex TINYINT ) UNIQUE KEY(user_id, user_name) DISTRIBUTED BY HASH(user_id) BUCKETS 10 PROPERTIES ( "enable_unique_key_merge_on_write" = "true" ); show create table ods_a.example_tbl_unique CREATE TABLE `example_tbl_unique` ( `user_id` largeint NOT NULL, `user_name` varchar(50) NOT NULL, `city` varchar(20) NULL, `age` smallint NULL, `sex` tinyint NULL ) ENGINE=OLAP UNIQUE KEY(`user_id`, `user_name`) DISTRIBUTED BY HASH(`user_id`) BUCKETS 10 PROPERTIES ( "file_cache_ttl_seconds" = "0", "is_being_synced" = "false", "storage_medium" = "hdd", "storage_format" = "V2", "inverted_index_storage_format" = "V2", "enable_unique_key_merge_on_write" = "true", "light_schema_change" = "true", "storage_vault_id" = "3", "storage_vault_name" = "hdfs_demo_a", "disable_auto_compaction" = "false", "enable_single_replica_compaction" = "false", "group_commit_interval_ms" = "10000", "group_commit_data_bytes" = "134217728", "enable_mow_light_delete" = "false" );

5.2.4.3 库设置a，表设置b时，且用户有hdfs_vault\hdfs_demo_a\hdfs_demo_b的权限时

CREATE DATABASE IF NOT EXISTS `ods_a` PROPERTIES ( "storage_vault_name" = "hdfs_demo_a" ); CREATE TABLE IF NOT EXISTS ods_a.example_tbl_unique_b ( user_id LARGEINT NOT NULL, user_name VARCHAR(50) NOT NULL, city VARCHAR(20), age SMALLINT, sex TINYINT ) UNIQUE KEY(user_id, user_name) DISTRIBUTED BY HASH(user_id) BUCKETS 10 PROPERTIES ( "enable_unique_key_merge_on_write" = "true", "storage_vault_name" = "hdfs_vault_b" ); show create table ods_a.example_tbl_unique_b CREATE TABLE `example_tbl_unique_b` ( `user_id` largeint NOT NULL, `user_name` varchar(50) NOT NULL, `city` varchar(20) NULL, `age` smallint NULL, `sex` tinyint NULL ) ENGINE=OLAP UNIQUE KEY(`user_id`, `user_name`) DISTRIBUTED BY HASH(`user_id`) BUCKETS 10 PROPERTIES ( "file_cache_ttl_seconds" = "0", "is_being_synced" = "false", "storage_medium" = "hdd", "storage_format" = "V2", "inverted_index_storage_format" = "V2", "enable_unique_key_merge_on_write" = "true", "light_schema_change" = "true", "storage_vault_id" = "4", "storage_vault_name" = "hdfs_demo_b", "disable_auto_compaction" = "false", "enable_single_replica_compaction" = "false", "group_commit_interval_ms" = "10000", "group_commit_data_bytes" = "134217728", "enable_mow_light_delete" = "false" );

5.2.4.4 库没设置，表没设置，用户没有hdfs_vault\hdfs_demo_a\hdfs_demo_b的权限时

（1）在ods库创建表时，报异常。没有hdfs_vault权限；

（2）在ods库查看表时，有表信息，可以读写数据；

（3）在ods_a库查看表时，有表信息，可以读写数据；

（4）在任何库，只要用户对这个库有drop权限就可以删除任意表；

5.2.5 授予/撤销权限

GRANT USAGE_PRIV ON STORAGE VAULT <vault_name> TO { ROLE | USER } {<role> | <user>} REVOKE USAGE_PRIV ON STORAGE VAULT <vault_name> FROM { ROLE | USER } {<role> | <user>}

6. 计算组

在存算分离架构下，可以将一个或多个计算节点 (BE) 组成一个计算组 (Compute Group)。本文档介绍如何使用计算组，其中涉及的操作包括：

• 查看所有计算组
• 计算组授权
• 在用户级别绑定计算组 (default_compute_group) 以达到用户级别的隔离效果

6.1 查看所有计算组

使用SHOW COMPUTE GROUPS命令可以查看当前仓库中的所有计算组。返回结果会根据用户权限级别显示不同内容：

• 具有ADMIN权限的用户可以查看所有计算组
• 普通用户只能查看其拥有使用权限（USAGE_PRIV）的计算组
• 如果用户没有任何计算组的使用权限，则返回结果为空

SHOW COMPUTE GROUPS;

6.2 计算组权限

6.2.1 授予计算组访问权限

前置条件：当前操作用户具备ADMIN权限，或者当前用户属于 admin role。

GRANT USAGE_PRIV ON COMPUTE GROUP {compute_group_name} TO {user};

6.2.2 撤销计算组访问权限

前置条件：当前操作用户具备ADMIN权限，或者当前用户属于 admin role。

GRANT USAGE_PRIV ON COMPUTE GROUP {compute_group_name} TO {user};

6.2.3 撤销计算组访问权限

前置条件：当前操作用户具备ADMIN权限，或者当前用户属于 admin role。

REVOKE USAGE_PRIV ON COMPUTE GROUP {compute_group_name} FROM {user};

6.2.4 设置默认计算组

为当前用户设置默认计算组（此操作需要当前用户已经拥有计算组的使用权限）：

SET PROPERTY 'default_compute_group' = '{clusterName}';

为其他用户设置默认计算组（此操作需要 Admin 权限）：

SET PROPERTY FOR {user} 'default_compute_group' = '{clusterName}';

查看当前用户默认计算组，返回结果中default_compute_group的值即为默认计算组：

SHOW PROPERTY;

查看其他用户默认计算组，此操作需要当前用户具备 admin 权限，返回结果中default_compute_group的值即为默认计算组：

SHOW PROPERTY FOR {user};

查看当前仓库下所有可用的计算组：

SHOW COMPUTE GROUPS;

6.2.5 没有任务一个计算组时