背景介绍
组学数据越来越大，数据传输已经成了一个组学合作研究中的重要限制因素。
百度云盘是数据存储和传输的重要工具，bypy是一个为了解决数据操作的工具。
它可以通过命令行上传和下载数据。
本文记录一下最近用该工具下载数据的应用。
安装
首先用conda建立一个环境：
module load miniconda
conda create --prefix bypy
source activate /path to bypy/bypy
conda install bypy
授权码的获取
运行：
bypy info
运行该命令会产生一个网址链接，把链接复制到浏览器中，检测到已登录的百度云盘的账户，而产生一个和账户授权码。
输入该授权码后就可以对该账户下的数据进行操作了。
该授权码输入一次后电脑下的操作就不用重复输入。
下载数据
注意：只能读取我的应用数据/bypy 文件夹下的数据。
运行：
bypy downfile
# 注意路径不要包含/apps/bypy/， 系统会自动加上。
下载文件夹可以用:
bypy downdir
更多功能
该工具有更多细致的功能，可以自行探索。
bypy --help
# 可以列出所有的功能选项
usage: bypy [-h] [-V] [-d] [-v] [-r RETRY] [-q] [-t TIMEOUT] [-s SLICE] [--chunk CHUNK] [-e] [-f] [--no-resume-download] [--include-regex INCREGEX] [--on-dup ONDUP]
[--no-symlink] [--disable-ssl-check] [--cacerts CACERTS] [--mirror MIRROR] [--select-fastest-mirror] [--rapid-upload-only]
[--resume-download-revert-back RCOUNT] [--move] [--processes PROCESSES] [--downloader DOWNLOADER] [--downloader-arguments DOWNLOADER_ARGS]
[--config-dir CONFIGDIR] [-c]
[command ...]
bypy v1.8.2 - Python client for Baidu Yun (Personal Cloud Storage) 百度云/百度网盘 Python 客户端
positional arguments:
command operations (quota, list, etc)
optional arguments:
-h, --help show this help message and exit
-V, --version show program's version number and exit
-d, --debug set debugging level (-dd to increase debugging level, -ddd to enable HTPP traffic debugging as well (very talkative)) [default: 0]
-v, --verbose set verbosity level [default: 0]
-r RETRY, --retry RETRY
number of retry attempts on network error [default: 5 times]
-q, --quit-when-fail quit when maximum number of retry failed [default: False]
-t TIMEOUT, --timeout TIMEOUT
network timeout in seconds [default: 300]
-s SLICE, --slice SLICE
size of file upload slice (can use '1024', '2k', '3MB', etc) [default: 20 MB]
--chunk CHUNK size of file download chunk (can use '1024', '2k', '3MB', etc) [default: 20 MB]
-e, --verify verify upload / download [default : False]
-f, --force-hash force file MD5 / CRC32 calculation instead of using cached value
--no-resume-download resume instead of restarting when downloading if local file already exists [default: True]
--include-regex INCREGEX
regular expression of files to include. if not specified (default), everything is included. for download, the regex applies to the remote files;
for upload, the regex applies to the local files. to exclude files, think about your regex, some tips here:
https://stackoverflow.com/questions/406230/regular-expression-to-match-string-not-containing-a-word [default: ]
--on-dup ONDUP what to do when the same file / folder exists in the destination: 'overwrite', 'skip', 'prompt' [default: overwrite]
--no-symlink DON'T follow symbol links when uploading / syncing up
--disable-ssl-check DON'T verify host SSL cerificate
--cacerts CACERTS Specify the path for CA Bundle [default: None]
--mirror MIRROR Specify the PCS mirror (e.g. bj.baidupcs.com. Open 'https://pcs.baidu.com/rest/2.0/pcs/manage?method=listhost' to get the list) to use. [default:
pcs.baidu.com]
--select-fastest-mirror
Let the program run some tests and select the fastest PCS mirror it detectes. [default: False]
--rapid-upload-only only upload large files that can be rapidly uploaded
--resume-download-revert-back RCOUNT
Revert back at least RCOUNT download chunk(s) and align to chunk boundary when resuming the download. A negative value means NO reverts. [default:
1]
--move Delete source files/directories after download/upload/syncdown/syncup is successful (This will force verification of the files). [default: False]
--processes PROCESSES
Number of parallel processes. (Only applies to dir sync/dl/ul). [default: 1]
--downloader DOWNLOADER
downloader to use (use python if not specified). valid values: ['aria2'] [default: ]
--downloader-arguments DOWNLOADER_ARGS
arguments for the downloader: normally, the string is the arguments to be passed to the downloader. however, when it begins with '@', it will be
treated as the name of file, whose contents will be used as the downloader arguments (example: when specifying '@args.txt', file contents of
'args.txt' will be used as the downloader arguments, not the string '@args.txt' itself). you can also use environment variable
'DOWNLOADER_ARGUMENTS' to specify the downloader arguments (the environment variable has lower priority compared to this argument). default values:
{'aria2': '-c -k10M -x4 -s4 --file-allocation=none'}
--config-dir CONFIGDIR
specify the config path [default: /home/zhenbin.hu/.bypy]
-c, --clean remove the token file (need re-auth) and upload progress file, -cc: clean hash cache file as well
Commands:
refreshtoken - refresh the access token
cdl_add <source_url> [save_path] [timeout] - add an offline (cloud) download task
cdl_addmon <source_url> [save_path] [timeout] - add an offline (cloud) download task and monitor the download progress
cdl_cancel - cancel an offline (cloud) download task
cdl_list - list offline (cloud) download tasks
cdl_query - query existing offline (cloud) download tasks
cleancache - remove invalid entries from hash cache file
combine <remotefile> [localfile] [md5s] - try to create a file at PCS by combining slices, having MD5s specified
compare [remotedir] [localdir] - compare the remote directory with the local directory
copy/cp <from> <to> - copy a file / dir remotely at Baidu Yun
delete/remove/rm <remotepath> - delete a file / dir remotely at Baidu Yun
downdir [remotedir] [localdir] - download a remote directory (recursively)
downfile <remotefile> [localpath] - download a remote file.
download [remotepath] [localpath] - download a remote directory (recursively) / file
dumpcache - display file hash cache
list/ls [remotepath] [format] [sort] [order] - list the 'remotepath' directory at Baidu PCS
listrecycle [start] [limit] - list the recycle contents
meta <remotepath> [format] - get information of the given path (dir / file) at Baidu Yun.
mkdir <remotedir> - create a directory at Baidu Yun
move/mv/rename/ren <from> <to> - move a file / dir remotely at Baidu Yun
quota/info - display the quota information
restore <remotepath> - restore a file from the recycle bin
search <keyword> [remotepath] [recursive] - search for a file using keyword at Baidu Yun
stream <remotefile> <localpipe> [format] [chunk] - stream a video / audio file converted to M3U format at cloud side, to a pipe.
syncdown [remotedir] [localdir] [deletelocal] - sync down from the remote directory to the local directory
syncup [localdir] [remotedir] [deleteremote] - sync up from the local directory to the remote directory
upload [localpath] [remotepath] [ondup] - upload a file or directory (recursively)
whoami - display the user information
