发表于 分类于: 阅读次数:

前端实现方式请查看另一篇文章：Minio直传方案之前端实现：Vue+Element+Axios实现分片上传至Minio。

前言

​ 一般的 minio 使用中，我们基本都是前端上传到服务器，再由服务器上传至 minio 平台。在上传小文件时并没有什么问题，但在上传大文件时就显得不稳定及上传效率低，因为要经过后台服务器中转。

​ 我在想如果不经过后台服务中转，由前端直接上传至 minio，但前端有需要存储密钥等连接信息，很不安全，那还有没有其他办法呢？

在 SpringBoot通过Minio实现大文件分片上传 看到相关解决方案。

由于 Java 版本的 Minio SDK 默认不允许单独调用分片的相关方法，我们需要自定义 Client 并继承 MinioClient 才可以使用分片方法。

流程：

  1. 前端获取文件MD5，发送至后台判断是否有该文件，有则直接转存；
  2. 前端调用初始化接口（分多少片协商好），后端调用 minio 初始化，返回分片上传地址和 uploadId；
  3. 前端上传分片文件；
  4. 上传完成后，前端发送请求至后台服务，后台服务调用 minio 合并文件；

流程图：

整体流程

实现

以下仅展示核心代码，完整代码请查看 GitHub 仓：spring-boot-minio 。

自定义 MinioClient

public class CustomMinioClient extends MinioClient {

    protected CustomMinioClient(MinioClient client) {
        super(client);
    }

    /**
     * 获取 uploadId
     * @param bucketName bucketName
     * @param region region
     * @param objectName objectName
     * @param headers headers
     * @param extraQueryParams extraQueryParams
     * @return
     * @throws ServerException
     * @throws InsufficientDataException
     * @throws ErrorResponseException
     * @throws NoSuchAlgorithmException
     * @throws IOException
     * @throws InvalidKeyException
     * @throws XmlParserException
     * @throws InvalidResponseException
     * @throws InternalException
     */
    public String getUploadId(String bucketName, String region, String objectName,
                              Multimap<String, String> headers, Multimap<String, String> extraQueryParams)
            throws ServerException, InsufficientDataException, ErrorResponseException, NoSuchAlgorithmException, IOException, InvalidKeyException, XmlParserException, InvalidResponseException, InternalException {
        CreateMultipartUploadResponse response = this.createMultipartUpload(bucketName, region, objectName, headers, extraQueryParams);

        return response.result().uploadId();
    }

    /**
     * 合并分片
     * @param bucketName
     * @param region
     * @param objectName
     * @param uploadId
     * @param parts
     * @param extraHeaders
     * @param extraQueryParams
     * @return
     * @throws ServerException
     * @throws InsufficientDataException
     * @throws ErrorResponseException
     * @throws NoSuchAlgorithmException
     * @throws IOException
     * @throws InvalidKeyException
     * @throws XmlParserException
     * @throws InvalidResponseException
     * @throws InternalException
     */
    public ObjectWriteResponse mergeMultipart(String bucketName, String region, String objectName, String uploadId,
                                              Part[] parts, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams)
            throws ServerException, InsufficientDataException, ErrorResponseException, NoSuchAlgorithmException, IOException, InvalidKeyException, XmlParserException, InvalidResponseException, InternalException {
        return this.completeMultipartUpload(bucketName, region, objectName, uploadId, parts,extraHeaders, extraQueryParams);
    }

    /**
     * 查询分分片列表
     * @param bucketName
     * @param region
     * @param objectName
     * @param maxParts
     * @param partNumberMaker
     * @param uploadId
     * @param extraHeaders
     * @param extraQueryParams
     * @return
     * @throws ServerException
     * @throws InsufficientDataException
     * @throws ErrorResponseException
     * @throws NoSuchAlgorithmException
     * @throws IOException
     * @throws InvalidKeyException
     * @throws XmlParserException
     * @throws InvalidResponseException
     * @throws InternalException
     */
    public ListPartsResponse listMultipart(String bucketName, String region, String objectName, Integer maxParts, Integer partNumberMaker,
                                           String uploadId, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws ServerException, InsufficientDataException, ErrorResponseException, NoSuchAlgorithmException, IOException, InvalidKeyException, XmlParserException, InvalidResponseException, InternalException {
        return this.listParts(bucketName, region, objectName, maxParts, partNumberMaker, uploadId, extraHeaders, extraQueryParams);
    }
}

Minio 配置

  1. minio 配置属性；
@Getter
@Setter
@Configuration
@ConfigurationProperties(prefix = "minio")
public class MinioPropertiesConfig {

    /**
     * 服务器 URL
     */
    private String endpoint;

    /**
     * Access key
     */
    private String accessKey;

    /**
     * Secret key
     */
    private String secretKey;

    /**
     * 默认的 bucket 名称
     */
    private String bucketName;

    /**
     * 运行上传的文件类型
     */
    private String allowFileType;

    /**
     * 分片上传有效期（单位：秒）
     */
    private Integer chunkUploadExpirySecond;

}
  1. 配置成 Spring Bean，使用时直接引入即可；
@Configuration
@EnableConfigurationProperties(MinioPropertiesConfig.class)
@Slf4j
public class MinioConfig {

    private MinioPropertiesConfig minioPropertiesConfig;

    @Autowired
    public void setMinioPropertiesConfig(MinioPropertiesConfig minioPropertiesConfig) {
        this.minioPropertiesConfig = minioPropertiesConfig;
    }

    @Bean
    public CustomMinioClient customMinioClient() {
        MinioClient minioClient;
        try {
            minioClient = MinioClient.builder()
                    .endpoint(minioPropertiesConfig.getEndpoint())
                    .credentials(minioPropertiesConfig.getAccessKey(), minioPropertiesConfig.getSecretKey())
                    .build();
        } catch (Exception e) {
            log.error("初始化 Minio 客户端失败：" + e.getMessage());
            throw e;
        }
        return new CustomMinioClient(minioClient);
    }

}
  1. 在 application.yml 中添加 minio 配置信息。
minio:
  endpoint: http://127.0.0.1:9000
  access-key: lanweihong
  secret-key: lanweihong
  bucket-name: test
  allow-file-type: jpg,png,jpeg,zip,rar,doc,docx,xls,xlsx,img,iso
  # 分片上传有效期: 秒
  chunk-upload-expiry-second: 86400

Minio 工具类

@Component
@Slf4j
public class MinioHelper {

    private final MinioPropertiesConfig minioPropertiesConfig;
    private final CustomMinioClient customMinioClient;

    @Autowired
    public MinioHelper(MinioPropertiesConfig minioPropertiesConfig, CustomMinioClient customMinioClient) {
        this.minioPropertiesConfig = minioPropertiesConfig;
        this.customMinioClient = customMinioClient;
    }

    /**
     * 初始化获取 uploadId
     * @param objectName 文件名
     * @param partCount 分片总数
     * @param contentType contentType
     * @return
     */
    public MinioUploadInfo initMultiPartUpload(String objectName, int partCount, String contentType) {
        HashMultimap<String, String> headers = HashMultimap.create();
        headers.put("Content-Type", contentType);

        String uploadId = "";
        List<String> partUrlList = new ArrayList<>();
        try {
            // 获取 uploadId
            uploadId = customMinioClient.getUploadId(minioPropertiesConfig.getBucketName(),
                    null,
                    objectName,
                    headers,
                    null);
            Map<String, String> paramsMap = new HashMap<>(2);
            paramsMap.put("uploadId", uploadId);
            for (int i = 1; i <= partCount; i++) {
                paramsMap.put("partNumber", String.valueOf(i));
                // 获取上传 url
                String uploadUrl = customMinioClient.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder()
                        // 注意此处指定请求方法为 PUT，前端需对应，否则会报 `SignatureDoesNotMatch` 错误
                        .method(Method.PUT)
                        .bucket(minioPropertiesConfig.getBucketName())
                        .object(objectName)
                        // 指定上传连接有效期
                        .expiry(minioPropertiesConfig.getChunkUploadExpirySecond(), TimeUnit.SECONDS)
                        .extraQueryParams(paramsMap).build());

                partUrlList.add(uploadUrl);
            }
        } catch (Exception e) {
            log.error("initMultiPartUpload Error:" + e);
           return null;
        }
        // 过期时间
        LocalDateTime expireTime = LocalDateTimeUtil.offset(LocalDateTime.now(), minioPropertiesConfig.getChunkUploadExpirySecond(), ChronoUnit.SECONDS);
        MinioUploadInfo result = new MinioUploadInfo();
        result.setUploadId(uploadId);
        result.setExpiryTime(expireTime);
        result.setUploadUrls(partUrlList);
        return result;
    }

    /**
     * 分片合并
     * @param objectName 文件名
     * @param uploadId uploadId
     * @return
     */
    public String mergeMultiPartUpload(String objectName, String uploadId) {
        // todo 最大1000分片 这里好像可以改吧
        Part[] parts = new Part[1000];
        int partIndex = 0;
        ListPartsResponse partsResponse = listUploadPartsBase(objectName, uploadId);
        if (null == partsResponse) {
            log.error("查询文件分片列表为空");
            throw new BusinessException("分片列表为空");
        }
        for (Part partItem : partsResponse.result().partList()) {
            parts[partIndex] = new Part(partIndex + 1, partItem.etag());
            partIndex++;
        }
        ObjectWriteResponse objectWriteResponse;
        try {
            objectWriteResponse = customMinioClient.mergeMultipart(minioPropertiesConfig.getBucketName(), null, objectName, uploadId, parts, null, null);
        } catch (Exception e) {
            log.error("分片合并失败：" + e);
            throw new BusinessException("分片合并失败：" + e.getMessage());
        }
        if (null == objectWriteResponse) {
            log.error("合并失败，合并结果为空");
            throw new BusinessException("分片合并失败");
        }
        return objectWriteResponse.region();
    }

    /**
     * 获取已上传的分片列表
     * @param objectName 文件名
     * @param uploadId uploadId
     * @return
     */
    public List<Integer> listUploadChunkList(String objectName, String uploadId) {
        ListPartsResponse partsResponse = listUploadPartsBase(objectName, uploadId);
        if (null == partsResponse) {
            return Collections.emptyList();
        }
        return partsResponse.result().partList().stream().map(Part::partNumber).collect(Collectors.toList());
    }

    private ListPartsResponse listUploadPartsBase(String objectName, String uploadId) {
        int maxParts = 1000;
        ListPartsResponse partsResponse;
        try {
            partsResponse = customMinioClient.listMultipart(minioPropertiesConfig.getBucketName(), null, objectName, maxParts, 0, uploadId, null, null);
        } catch (ServerException | InsufficientDataException | ErrorResponseException | NoSuchAlgorithmException | IOException | XmlParserException | InvalidKeyException | InternalException | InvalidResponseException e) {
            log.error("查询文件分片列表错误：{}，uploadId:{}", e, uploadId);
            return null;
        }
        return partsResponse;
    }
}

Service 层调用

@Service
@Slf4j
public class FileServiceImpl implements IFileService {

    private final MinioHelper minioHelper;
    private final IMinioFileUploadInfoService minioFileUploadInfoService;
    private final IMinioFileChunkUploadInfoService minioFileChunkUploadInfoService;

    @Autowired
    public FileServiceImpl(MinioHelper minioHelper, IMinioFileUploadInfoService minioFileUploadInfoService, IMinioFileChunkUploadInfoService minioFileChunkUploadInfoService) {
        this.minioHelper = minioHelper;
        this.minioFileUploadInfoService = minioFileUploadInfoService;
        this.minioFileChunkUploadInfoService = minioFileChunkUploadInfoService;
    }

    @Override
    public MinioUploadInfo getUploadId(GetMinioUploadInfoParam param) {
        // 计算分片数量
        double partCount = Math.ceil(param.getFileSize() / param.getChunkSize());
        MinioUploadInfo uploadInfo = minioHelper.initMultiPartUpload(param.getFileName(), (int) partCount, param.getContentType());
        return uploadInfo;
    }

    @Override
    public MinioOperationResult checkFileExistsByMd5(String md5) {
        MinioOperationResult result = new MinioOperationResult();
        MinioFileUploadInfoDTO minioFileUploadInfo = this.minioFileUploadInfoService.getByFileMd5(md5);
        if (null == minioFileUploadInfo) {
            result.setStatus(CommonEnums.MinioFileStatusEnum.UN_UPLOADED.ordinal());
            return result;
        }
        // 已上传
        if (minioFileUploadInfo.getFileStatus() == CommonEnums.MinioFileStatusEnum.UPLOADED.ordinal()) {
            result.setStatus(CommonEnums.MinioFileStatusEnum.UPLOADED.ordinal());
            result.setUrl(minioFileUploadInfo.getFileUrl());
            return result;
        }
        // 查询已上传分片列表并返回已上传列表
        List<Integer> chunkUploadedList = listUploadParts(minioFileUploadInfo.getFileName(), minioFileUploadInfo.getUploadId());
        result.setStatus(CommonEnums.MinioFileStatusEnum.UPLOADING.ordinal());
        result.setChunkUploadedList(chunkUploadedList);
        return result;
    }

    @Override
    public List<Integer> listUploadParts(String objectName, String uploadId) {
        return minioHelper.listUploadChunkList(objectName, uploadId);
    }

    @Override
    public String mergeMultipartUpload(MergeMinioMultipartParam param) {
        // 合并文件分片
        String result = minioHelper.mergeMultiPartUpload(param.getFileName(), param.getUploadId());
        if (!StringUtils.isEmpty(result)) {
            MinioFileUploadInfoParam fileUploadInfoParam = new MinioFileUploadInfoParam();
            fileUploadInfoParam.setFileUrl(result);
            fileUploadInfoParam.setFileMd5(param.getMd5());
            fileUploadInfoParam.setFileStatus(CommonEnums.MinioFileStatusEnum.UPLOADED.ordinal());

            // 更新状态
            minioFileUploadInfoService.updateFileStatusByFileMd5(fileUploadInfoParam);
        }
        return result;
    }
}

Controller

@RestController
public class MinioController {

    private final IFileService fileService;

    @Autowired
    public MinioController(IFileService fileService) {
        this.fileService = fileService;
    }

    /**
     * 获取上传 url
     * @param param 参数
     * @return
     */
    @PostMapping("/upload")
    public JsonResult<MinioUploadInfo> getUploadId(@Validate @RequestBody GetMinioUploadInfoParam param) {
        MinioUploadInfo minioUploadId = fileService.getUploadId(param);
        return JsonResult.ok(minioUploadId);
    }

    /**
     * 校验文件是否存在
     * @param md5 文件 md5
     * @return
     */
    @GetMapping("/upload/check")
    public JsonResult<MinioOperationResult> checkFileUploadedByMd5(@RequestParam("md5")String md5) {
        MinioOperationResult result = fileService.checkFileExistsByMd5(md5);
        return JsonResult.ok(result);
    }

    /**
     * 合并文件
     * @param param
     * @return
     */
    @PostMapping("/upload/merge")
    public JsonResult<JSONObject> mergeUploadFile(@Valid MergeMinioMultipartParam param) {
        String result = fileService.mergeMultipartUpload(param);
        if (StringUtils.isEmpty(result)) {
            return JsonResult.error("合并失败");
        }
        JSONObject object = new JSONObject();
        object.set("url", result);
        return JsonResult.ok(object);
    }
}

总结

​ 本文仅介绍使用流程及简单的实现，整体流程较简单，代码已上传至 GitHub：spring-boot-minio。项目代码比较简单，目前未结合业务做整体流程实现，仅简单实现上传流程，但也具有一定的参考性，请勿直接用于生产环境，请根据自身需求优化

参考文献

[1]. SpringBoot通过Minio实现大文件分片上传

文章目录
  1. 前言
  2. 实现
    1. 自定义 MinioClient
    2. Minio 配置
    3. Minio 工具类
    4. Service 层调用
    5. Controller
  3. 总结
  4. 参考文献