昇腾平台适配GENERator模型实践
·
作者：昇腾实战派
知识地图：https://blog.csdn.net/Lumos_Lovegood/article/details/161601003
背景概述
GENERator是一个基于Transformer架构的基因组语言模型，可用于DNA序列理解、功能预测等生物信息学任务。本文记录了在Atlas 800I A3推理服务器上，从零搭建GENERator运行环境并完成NPU适配的全过程，包括Conda虚拟环境创建、依赖安装、CANN与PyTorch NPU插件配置，以及常见问题的排查与解决。旨在为后续在昇腾平台上运行类似基因组大模型提供可复现的参考。
一、环境准备
部署环境：Atlas 800I A3
HDK：25.5.0
CANN：8.3.RC1
Python：3.10.19
PyTorch：2.8.0+cpu
torch-npu：2.8.0
1. 创建conda虚拟环境并进入
conda create -n GENERator python=3.10 -y
conda activate GENERator
2. 克隆项目
参考官方仓库：https://github.com/GenerTeam/GENERator
git clone https://github.com/GenerTeam/GENERator.git
cd GENERator
3. 安装依赖
pip install -r requirements.txt
pip install liger-kernel
pip install flash-attn --no-build-isolation
安装 flash-attn 时若出现编译错误，可暂时忽略，不影响后续主要功能。
4. 安装CANN
从昇腾社区下载以下安装包：
- Ascend-cann-toolkit_8.2.RC1_linux-aarch64.run
- Ascend-cann-kernels-910b_8.2.RC1_linux-aarch64.run
上传至服务器后，按官方文档安装，并执行：
source /usr/local/Ascend/ascend-toolkit/set_env.sh
5. 安装torch与torch_npu
1）查看当前CANN版本，选择对应torch_npu版本
参考官方兼容性列表：https://www.hiascend.com/document/detail/zh/Pytorch/710/configandinstg/instg/insg_0004.html
cat /usr/local/Ascend/ascend-toolkit/latest/version.cfg
根据版本号下载对应的torch与torch_npu，并按安装命令执行。
2）验证安装
python3 -c "import torch;import torch_npu; a = torch.randn(3, 4).npu(); print(a + a);"
若输出类似以下信息，则说明PyTorch与NPU插件安装成功：
tensor([[-2.9474, -0.5735, -3.1606, -0.2197],
[-2.8293, 2.0746, -1.6005, -1.1916],
[-0.4026, -2.3322, 1.9780, -2.0485]], device='npu:0')
3）常见问题
问题：ImportError: cannot import name 'AttrsDescriptor' from 'triton.backends.compiler'
原因：安装torch时CANN版本不匹配。例如，在CANN 8.2.RC1环境下误装了对应8.3.RC1的torch_npu。解决方案：重新选择与当前CANN版本匹配的torch_npu版本（如CANN 8.3.RC1对应torch_npu 2.7.0系列），卸载后重新安装即可。
6. 指定使用NPU卡
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 # Atlas 800I A3 上对应4卡8die
二、代码适配NPU
1. 在运行脚本开头添加torch_npu导入
对于使用PyTorch的模型，在相应脚本文件开头添加以下两行即可完成迁移适配：
import torch_npu
from torch_npu.contrib import transfer_to_npu
transfer_to_npu 会自动将 torch.cuda 的大部分算子替换为 torch_npu.npu，实现无缝迁移。
2. 移除cuda相关代码
运行脚本时若出现与 torch.cuda 相关的报错，可将对应行注释掉。sequence_understanding.py 和 fine_tuning.py 中均涉及此类代码。
三、模型脚本运行
1. Causal Language Modeling Fine-tuning
脚本说明
各参数含义如下：
--model_name：模型名称，例如GenerTeam/GENERator-eukaryote-1.2b-base--dataset_name：数据集名称，例如GenerTeam/DeepSTARR-enhancer-activity--batch_size：批大小--num_train_epochs：训练轮数
执行示例：
python src/tasks/downstream/fine_tuning.py \
--model_name GenerTeam/GENERator-eukaryote-1.2b-base \
--dataset_name GenerTeam/DeepSTARR-enhancer-activity \
--batch_size 128 \
--num_train_epochs 1
2. Sequence Recovery
1）执行脚本
python src/tasks/downstream/sequence_recovery.py \
--bf16 \
--model_path GenerTeam/GENERator-v2-eukaryote-1.2b-base
2）执行结果
3）遇到的问题
报错KeyError: 'hash_index'
解决方案：升级tranformers后，正常
3. Sequence Understanding (Classification/Regression)
1）执行脚本
#单机多卡，跑 Enhancer Activity
torchrun --nnodes=1 \
--nproc_per_node=8 \
--rdzv_backend=c10d \
src/tasks/downstream/sequence_understanding.py \
--dataset_name GenerTeam/DeepSTARR-enhancer-activity \
--problem_type regression \
--model_name GenerTeam/GENERator-v2-eukaryote-1.2b-base
注意，模型默认跑的是"GenerTeam/GENERator-eukaryote-1.2b-base"，如果想跑其他模型需要添加–model_name参数
2）运行结果
***** Running Evaluation *****
Num examples = 40570
Batch size = 16
{'eval_loss': 0.4794687330722809, 'eval_mse_label_0': 0.5370198488235474, 'eval_mae_label_0': 0.5627561211585999, 'eval_r2_label_0': 0.4606979489326477, 'eval_pearson_label_0': 0.6828397512435913, 'eval_mse_label_1': 0.4218886196613312, 'eval_mae_label_1': 0.49110424518585205, 'eval_r2_label_1': 0.5980653762817383, 'eval_pearson_label_1': 0.7739343047142029, 'eval_mse': 0.47945424914360046, 'eval_mae': 0.5269302725791931, 'eval_r2': 0.5313684940338135, 'eval_pearson': 0.7306988835334778, 'eval_runtime': 12.6611, 'eval_samples_per_second': 3204.293, 'eval_steps_per_second': 25.037, 'epoch': 2.31}
0%|▍ | 7245/3143000 [53:14<302:15:49, 2.88it/sSaving model checkpoint to results/sequence_understanding/checkpoint-7245
Configuration saved in results/sequence_understanding/checkpoint-7245/config.json
Model weights saved in results/sequence_understanding/checkpoint-7245/model.safetensors
tokenizer config file saved in results/sequence_understanding/checkpoint-7245/tokenizer_config.json
Special tokens file saved in results/sequence_understanding/checkpoint-7245/special_tokens_map.json
Deleting older checkpoint [results/sequence_understanding/checkpoint-6615] due to args.save_total_limit
{'loss': 0.2591, 'grad_norm': 11.105179786682129, 'learning_rate': 6.25e-07, 'epoch': 2.31}
{'loss': 0.2568, 'grad_norm': 5.865702152252197, 'learning_rate': 6.25e-07, 'epoch': 2.31}
{'loss': 0.2673, 'grad_norm': 6.335740089416504, 'learning_rate': 6.25e-07, 'epoch': 2.31}
{'loss': 0.281, 'grad_norm': 5.783288478851318, 'learning_rate': 6.25e-07, 'epoch': 2.32}
{'loss': 0.2761, 'grad_norm': 5.740670204162598, 'learning_rate': 6.25e-07, 'epoch': 2.32}
{'loss': 0.2638, 'grad_norm': 6.165787696838379, 'learning_rate': 6.25e-07, 'epoch': 2.32}
{'loss': 0.2745, 'grad_norm': 4.575372695922852, 'learning_rate': 6.25e-07, 'epoch': 2.33}
{'loss': 0.2803, 'grad_norm': 11.821925163269043, 'learning_rate': 6.25e-07, 'epoch': 2.33}
{'loss': 0.2662, 'grad_norm': 5.751617431640625, 'learning_rate': 6.25e-07, 'epoch': 2.33}
{'loss': 0.2852, 'grad_norm': 8.808683395385742, 'learning_rate': 6.25e-07, 'epoch': 2.34}
{'loss': 0.2831, 'grad_norm': 5.9474687576293945, 'learning_rate': 6.25e-07, 'epoch': 2.34}
{'loss': 0.2822, 'grad_norm': 10.884577751159668, 'learning_rate': 6.25e-07, 'epoch': 2.34}
{'loss': 0.272, 'grad_norm': 7.912357807159424, 'learning_rate': 6.25e-07, 'epoch': 2.34}
{'loss': 0.2737, 'grad_norm': 6.498815059661865, 'learning_rate': 6.25e-07, 'epoch': 2.35}
{'loss': 0.2751, 'grad_norm': 9.954439163208008, 'learning_rate': 6.25e-07, 'epoch': 2.35}
{'loss': 0.2757, 'grad_norm': 6.915566921234131, 'learning_rate': 6.25e-07, 'epoch': 2.35}
{'loss': 0.2358, 'grad_norm': 8.045971870422363, 'learning_rate': 6.25e-07, 'epoch': 2.36}
{'loss': 0.2753, 'grad_norm': 11.633805274963379, 'learning_rate': 6.25e-07, 'epoch': 2.36}
{'loss': 0.2765, 'grad_norm': 9.192776679992676, 'learning_rate': 6.25e-07, 'epoch': 2.36}
{'loss': 0.2787, 'grad_norm': 7.927358150482178, 'learning_rate': 6.25e-07, 'epoch': 2.37}
{'loss': 0.2639, 'grad_norm': 5.664511203765869, 'learning_rate': 6.25e-07, 'epoch': 2.37}
{'loss': 0.2717, 'grad_norm': 8.179108619689941, 'learning_rate': 6.25e-07, 'epoch': 2.37}
{'loss': 0.2742, 'grad_norm': 6.19037389755249, 'learning_rate': 6.25e-07, 'epoch': 2.38}
{'loss': 0.2615, 'grad_norm': 6.295119762420654, 'learning_rate': 6.25e-07, 'epoch': 2.38}
{'loss': 0.2657, 'grad_norm': 7.873654842376709, 'learning_rate': 6.25e-07, 'epoch': 2.38}
{'loss': 0.2693, 'grad_norm': 10.18658447265625, 'learning_rate': 6.25e-07, 'epoch': 2.39}
{'loss': 0.2633, 'grad_norm': 7.398651599884033, 'learning_rate': 6.25e-07, 'epoch': 2.39}
{'loss': 0.2638, 'grad_norm': 7.963374614715576, 'learning_rate': 6.25e-07, 'epoch': 2.39}
{'loss': 0.27, 'grad_norm': 10.287927627563477, 'learning_rate': 6.25e-07, 'epoch': 2.4}
{'loss': 0.2721, 'grad_norm': 11.135351181030273, 'learning_rate': 6.25e-07, 'epoch': 2.4}
{'loss': 0.2525, 'grad_norm': 5.727623462677002, 'learning_rate': 6.25e-07, 'epoch': 2.4}
{'loss': 0.2853, 'grad_norm': 13.691278457641602, 'learning_rate': 6.25e-07, 'epoch': 2.41}
0%|▍ | 7560/3143000 [55:17<273:23:12, 3.19it/s]
***** Running Evaluation *****
Num examples = 40570
Batch size = 16
{'eval_loss': 0.48049435019493103, 'eval_mse_label_0': 0.5383994579315186, 'eval_mae_label_0': 0.5640773177146912, 'eval_r2_label_0': 0.45931243896484375, 'eval_pearson_label_0': 0.6814720034599304, 'eval_mse_label_1': 0.42255616188049316, 'eval_mae_label_1': 0.4916130304336548, 'eval_r2_label_1': 0.5974294543266296, 'eval_pearson_label_1': 0.7738450169563293, 'eval_mse': 0.48047783970832825, 'eval_mae': 0.5278451442718506, 'eval_r2': 0.5303679704666138, 'eval_pearson': 0.7301602959632874, 'eval_runtime': 12.8483, 'eval_samples_per_second': 3157.615, 'eval_steps_per_second': 24.673, 'epoch': 2.41}
0%|▍ | 7560/3143000 [55:30<273:23:12, 3.19it/sSaving model checkpoint to results/sequence_understanding/checkpoint-7560
/root/miniconda3/envs/GENERator/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
/root/miniconda3/envs/GENERator/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
/root/miniconda3/envs/GENERator/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
/root/miniconda3/envs/GENERator/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
/root/miniconda3/envs/GENERator/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
/root/miniconda3/envs/GENERator/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
/root/miniconda3/envs/GENERator/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
Configuration saved in results/sequence_understanding/checkpoint-7560/config.json
Model weights saved in results/sequence_understanding/checkpoint-7560/model.safetensors
tokenizer config file saved in results/sequence_understanding/checkpoint-7560/tokenizer_config.json
Special tokens file saved in results/sequence_understanding/checkpoint-7560/special_tokens_map.json
Deleting older checkpoint [results/sequence_understanding/checkpoint-6930] due to args.save_total_limit
Training completed. Do not forget to share your model on huggingface.co/models =)
/root/miniconda3/envs/GENERator/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
Loading best model from results/sequence_understanding/checkpoint-5985 (score: 0.46442121267318726).
{'train_runtime': 3471.4241, 'train_samples_per_second': 115887.887, 'train_steps_per_second': 905.392, 'train_loss': 0.4459437266859428, 'epoch': 2.41}
0%|▍ | 7560/3143000 [55:51<386:03:56, 2.26it/s]
✅ Training completed in 57.89 minutes
作为“人工智能6S店”的官方数字引擎，为AI开发者与企业提供一个覆盖软硬件全栈、一站式门户。
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)