Release Notes¶
JuiceFS 4.4.9 (2020/11/16)¶
New
Released a tool to migrate data from HDFS to JuiceFS gracefully.
Support IBM COS.
Change
Interrupt write operations in compatible mode.
Improve performance when librados was used to talk to Ceph.
Follows symlink when deleting files or directories using Hadoop SDK.
Bugfix
Fixed a bug when listing a directory with symlinks.
Fixed a bug in Hadoop SDK when file is closed twice.
JuiceFS 4.4.8 (2020/10/21)¶
New
Use JuiceFS as a transparent caching lay for HDFS.
Change
Added checksum for meta requests.
How used space for mounted subdir instead of whole volume.
Support symlink for all operations in Hadoop SDK.
Improve date locality when cache shareing is enabled.
Updated /etc/mtab with Go FUSE.
Bugfix
Fixed a bug that may cause crash during write operation.
Fixed a bug that may cause POSIX lock to fail.
JuiceFS 4.4.7 (2020/09/18)¶
New
A HTTP gateway compatible with S3 was beta released, which works with POSIX and Hadoop SDK.
The HTTP API embeded in client is compatible with WebDAV.
The bandwidth to object storage used by all clients can be controlled.
The FUSE module in Go was enabled by default, could be disabled using GOFUSE=0.
Change
Improved the performance of concurrent reading on the same big file.
Limit cached metadata to 5 millions of inodes (up to 5 minutes).
Dump all the goroutines once fail to flush data.
Limit getattr(1) and statvfs to 300ms, so df will not block.
Bugfix
Fixed the problem that initialization fail always once interruptted.
JuiceFS 4.4.6 (2020/08/14)¶
New
Added faster FUSE module (in Go), which could be enabled by GOFUSE=1.
Support copy_file_range() to allow faster copy.
Added a parameter to control the number of threads to prefetch caching blocks.
Supports MIPS64EL (Loongsun CPU).
Added performamce counters in .stats.
Supports internal endpoint for Qiniu.
Change
Show used space as 64KB instead of 0 when it’s less than 64KB.
Avoid crash when running out of memory.
Check usage of inodes in caching disk to avoid running out.
Show name of volume in macOS Finder.
Free space earlier for removed blocks.
Bugfix
Fixed a bug with metadata cache, wrong attributes for resued inodes after 24 hours.
Fixed a compatibility issue between Presto and Hadoop SDK.
Fixed idempotency of putting a object to Qiniu.
Fixed mounting with GCS.
JuiceFS 4.4.5 (2020/07/2)¶
New
Windows client supports case insensitivity.
Hadoop SDK support symlink to other system, for example, hdfs://nm1:8080/xxxx
Added a configuration for grouping in Hadoop SDK (refreshed in every 2 minutes).
Changes
Show total of lengths instead of billing size in hadoop fs -du (same as HDFS).
Check permission of AK/SK with prefix during mounting.
Bugfix
Fixed IO exception when read failed.
Fixed potential deadlock in compatible mode.
JuiceFS 4.4.4 (2020/06/14)¶
New
Added –ioretries to specify number of retry after metadata failure.
Added –cache-partial-only to only cache blocks for small files or partial read.
Changes
Improved performance of random write.
Improved performace of compacting.
Reduced the number of PUT requests when append on multiple files by combining them together.
Bugfix
Fixed the memory issue when compacting is too slow to catch up.
Fixed a issue that Hadoop SDK can not fetch node lists from YARN.
Fixed stale cache after inode is removed and reused again.
JuiceFS 4.4.3 (2020/05/20)¶
New
Windows client is available to enterprise customers，which supports Windows Server 2012、Windows 10.
Supports ARM64.
Supports caching data in memory, which could be enabled by –cache-dir memory.
Hadoop SDK supports p2p caching.
Changes
Random read support on p2p caching.
Improved performance for random read.
Bugfix
Fixed the problem that wrong data could be returned from Aliyun OSS when range is overflow.
Fixed the partial listing when empty result returned from Aliyun OSS bucket with multiple versions enabled.
Fixed a bug that backend is overloaded when many clients try to cleanup fail parts in the same time.
Fixed a compatibility issue with CDH 5.8.
JuiceFS 4.4.2 (2020/04/22)¶
Bugfix
Fixed the bug that could cause loss data during compacting under heavy write in writeback mode. The service is patched to workaround that by disallow compaction on buggy versions, which are 4.3.14, 4.4.0, 4.4.1 (build between 2/28/2020 and 4/09/2020). Please upgrade to 4.4.2 if you use writeback mode.
JuiceFS 4.4.1 (2020/04/12)¶
New
Supported TiKV as object storage.
Added a rule to disallow background job for some clients (see access control in web console).
Supported LZ4 as compression algorithm, faster than Zstandard, but lower compression ratio.
Supported all flags for fallocate.
Changes
Improved performance for sequential read on large files.
Improved performance in writeback mode.
Changed the algorithm to distribute the cache blocks within multiple caching directories.
Show PID in the logging.
Add a configuration to change the replica of cached blocks.
Bugfix
Fixed a bug that wildcast can only match some caching directories.
Fixed a bug in Hadoop SDK that fails to list files with more than 127 characters in the name.
Fixed confusing error message after read/write failure in Hadoop SDK.
JuiceFS 4.4.0 (2020/03/12)¶
New
Added –subdir to specify the subdirectory to mount (default: /).
Added –buffer-size to specify the memory used by read/write buffers, it’s 300MB by default.
Added –free-space-ratio to specified the minimum free space for caching disk.
Added arguments to disable POSIX or BSD lock.
Change
Huge improvements on readahead and memory management, memory usage decreased signafically.
Added retries for write to reduce IO Error.
Improved performance for disk cache.
Bugfix
Fixed a bug that could cause data corrupted under a random read pattern in Hadoop SDK .
JuiceFS 4.3.14 (2020/02/20)¶
New
Change the storage format of existing filesystem (compatible or optimized).
Added benchmark command (juicefs benchmark).
Change
Reduced memory usage when writing files.
Bugfix
Fixed the problem that warmup command that it could not warm up the whole big file.
Fixed the problem that it fail to create a GCS bucket when mounting for first time.
JuiceFS 4.3.13 (2020/01/20)¶
New
Added –max-space for mount command to limit the capacify of mount point.
Added –put-timeout to specify the timeout to upload a object, which will be retried.
Added a configuration (juicefs.superuser) for Hadoop SDK to specify the super user.
Change
Changed UserAgent to JuiceFS for requests to Tencent Cloud COS.
Supported Amazon S3 bucket with . in the name.
Support listing a directory with more than 10 millions of files.
No more syslog from Hadoop SDK.
Bugfix
Fixed the bug that posixlock/flock may fail with EIO.
Fixed the bug that the cached nlink is not invalidated properly.
Fixed that problem that temporate token for Aliyun OSS is not refreshed properly.
Fixed that problem that mtime is not updated properly in Hadoop SDK.
JuiceFS 4.3.12 (2019/11/14)¶
New
Supports POSIX ACL of Hadoop File System.
Supports file checksum in Hadoop File System, which is compatible with HDFS, allows DistCp to compare the checksum of files in JFS againt HDFS. It should be enabled by juicefs.file.checksum = true.
Added statistics for operations in Java client.
Change
Speedup rename() of Java client, only one RPC needed in common case.
Improved memory efficiency when read many files.
Bugfix
Fixed a bug which may hang the mountpoint during poor network condition.
Fixed a bug which cause wrong file syze for imported objects (since 4.3.11).
JuiceFS 4.3.11 (2019/10/17)¶
New
Supports Beansdb.
Supports extended attributes in Hadoop connector.
Cleanups files in Hadoop trash based on trash time.
Added –get-timeout to specify the max number of seconds to download a object.
Change
Improved the performance when metadata is not cached yet.
Interrupt timeouted read/write operations in close.
Added checksum for S3/OSS/COS for better data intergraty.
Allows more data (unlimited) to be cached in writeback mode.
Slowdown deletions to avoid spikes.
Create caching directories (matched with wildcast), could be turned off in Java client by juicefs.auto-create-cache-dir=false.
Cleanup cached blocks when free space ratio is low than 20%.
Bugfix
Fixed a bug that causes uid was mapped twice in snapshot/rmr.
Fixed a bug that causes block can not be recovered correctly.
JuiceFS 4.3.10 (2019/9/2)¶
New
Allows concurrent write on same file from multiple clients.
Allows use JuiceFS SDK with any schema, for example, s3://volume/path .
Added –opencache to speed up open operation.
For replicated volumes, writes will failover to secondary object storage in case of primary failure.
The operations are idempotent during retring after failover.
Change
Improved the consistence guarantee for metadata cache.
Improved write performance for compatible mode and random read performance for big files.
Improved the compatibility and performance for JuiceFS SDK.
Improved the performance when access JuiceFS via mountpoint in Hadoop (with JuiceFS SDK).
Bugfix
Fixed the problem that data could be corrupted if there are multiple flush during writes in compatible mode.
Fixed the problem that Digital Ocean bucket can’t be created during mounting.
JuiceFS 4.3.9 (2019/6/24)¶
New:
Hadoop client for JuiceFS is released as beta, is compatible with all Hadoop components.
New profiling tool for JuiceFS: juicefs profile.
Change:
Improve write performance by reducing RPC calls to metadata service.
Discover newly created user/group on client host.
JuiceFS 4.3.7 (2019/5/17)¶
New:
Supports using multiple directories for disk cache, by putting them in
--cache-dir(separate by:).Supports kernel writeback cache with
-o writeback_cache(kernel 3.15+ required), which can boost the performance for random write.Supports Wasabi.
Change:
Improved performance for random write (merge continious write together).
Change the fstype from fuse to fuse.juicefs, so we can stop mlocate from indexing files in JuiceFS by put fuse.juicefs into PRUNEFS.
Improved metadata cache for better performance.
Improved readahead for better reading performance.
Improved connection pool for OSS.
Reduces the HEAD request during GC.
Decreases the parallelism of compaction and synconization, reduces memory usage.
Bugfix:
Fixed a bug when importing a single object.
Added a workaround for kernel 3.1- which does not unflock the file before release.
JuiceFS 4.3.6 (2019/4/1)¶
New:
Supports Huawei Cloud OBS.
Supports Minio as object storage.
Bugfix:
Fixed the counter of packets sent to metadata service.
Fixed the problem that failed retry (fixed in 4.3.1) of appending causes broken file during failover. It’s recommanded to upgrade to latest version if the client is older than 4.3.1 .
JuiceFS 4.3.5 (2019/2/20)¶
New：
Supports 32-bit Linux。
Added lsof: List open files in last 10 minutes。
Added warmup: Download the blocks for all files in a directory in parallel, and fill them into local cache for future use.
Support Aliyun RAM role for ECS: Once a RAM role with OSS permissions attached to ECS, no AK is required during mounting (use empty string instead)
Change：
Improved writing performance in writeback mode (upload all the blocks in background).
Improved reading performance for small files when metadata cache is enabled.
Bugfix：
Fixed cache space overflow in writeback mode (–writeback).
Fixed permission error when metadata cache is enabled (–metacache)。
Fixed the IO error in batch mode (–batch).
JuiceFS 4.3.3 (2018/11/20)¶
New：
Added new options to tune meta cache in kernel：–attrcacheto, –entrycacheto, –direntrycacheto。
Allow remember passphrase (could be empty) using juicefs auth –passphrase.
Change：
Improved counters of API calls in .stats, more acurate.
Remove the floating-point encoding for size of directory in command line (no upgrade required).
Added build date and git revision to version string.
Allow disable cache using –cache-size 0.
Disable the watch dog who checks the healthy of mount point.
Disable HTTP/2 for Azure to increase parallism.
Bugfix：
Allow access Qiniu with HTTP (default is HTTPS).
Fixed the problem which cause client crash when cleaning fragments in compatible mode.
Fixed problem with concurrent cache loading。
Fixed the AppID for COS.
Fixed juicefs grep -v for inverse match.
JuiceFS 4.3.2 (2018/10/07)¶
New
Configurable compresssion：Compressed by Zstandard by default, could be turned off for better random read performance.
Expose client metrics using APIs：The metrics (/.stats) from clients are exposed in JSON API and Prometheus API.
New metrics in clients: read_bytes、write_bytes、uptime、cpuusage、memusage
Added Append Only accessing rule: Allow read and write new data, deletion and modification on existing data are disallowed.
JuiceFS 4.3.1 (2018/09/14)¶
New:
Configurable block size: default is 4MB, could be configured to 64K - 16M. For example, a smaller block size could improve the performance for random read (lower latency and higher IOPS).
Change：
Improved performance and stability of mounting, especially when the DNS is slow or non-repsonsive.
Improved support for KSYun, which can create the KS3 bucket as requested.
Improved the error messages when invalid command is provided.
Bugfix：
Fixed the bug that the upgrade version is not showed properly.
Fixed the duration between retry of connecting, which may cause IOError before.
Fixed the bug when JuiceFS is mounted to a relative path.
JuiceFS 4.3.0 (2018/08/22)¶
New：
Data encryption: Objects are encrypted using AES-256-GCM with a random generated key. The key is encrypted by a RSA private key and stored together with encrypted objects. RSA private key could be managed separately, or by Juicedata (protected by passphrase).
Added argument to limit the QPS of deletion to object store.
Added argument to limit the bandwidth of writing to object store.
Support Baidu Cloud.
Support Digital Ocean.
Changes：
juicefs will not upgrade itself automatically, should be done manually by juicefs version –upgrade.
Support FUSE fallocate(): –fallocate=none is not needed when benchmarking with fio.
Speedup importing: Added inode cache, skip imported objects faster.
Bugfix：
For busy read/write/truncate operations, stale length may cause IO error.
IO error when decompressing objects with unpected more data.
FD leak when concurrent open() in compatible mode.
limit of LIST request for Tencent COS is over 1000.
High CPU usage when localhost can’t be resolved to 127.0.0.1.
the tailing / can’t treat correctly.
JuiceFS 4.2.7 (2018/03/24)¶
New：
Add stats for List operations.
Support hardlink in compatible mode.
Allow modification on existing files in compatible mode (slow)。
Changes:
Disabled trash in compatible mode, which may cause data lose.
Improved performance on listing huge directory.
Wait for completion of file deletion in compatible mode.
Improved memory efficiency。
Bugfix：
Fixed availble space and inodes in vfsstat.
Fixed the pemission checking for rmr by non-root.
Fixed a potential deadlock when busy writing.
JuiceFS 4.2.6 (2018/03/05)¶
New：
Add compatible mode, files will be saved as single object with the full path as key, so they could be accessed using the RESTful API.
Small files could be combined into larger object to save PUT requests. This could be enabled by –writeback –batch.
Support Backblaze B2.
Changes：
When run in foreground (-f)，DEBUG log will not go into syslog.
JuiceFS 4.2.5 (2018/01/31)¶
New:
Remove all files and subdirectories quickly, see juicefs rmr for details.
Add a option to serve files using HTTP when mounting.
Add statistics for object storage accessing into .stats
Improvements：
Improve performance and reduce memory usage.
JuiceFS 4.2.4 (2017/12/10)¶
New:
Global Mirroring: we could help user to setup read-only mirror of JuiceFS in any regions of cloud provider, they have near real-time copy of all data (delay in seconds).
Enabled cache sharing within group. A group name could be specified when mounting (–group-name), all the clients with the same group name will share cache with each other, greatly improve the performance for hot dataset.
Enabled Web UI for trash, user could restore the files in trash from web console, or clean them up to release space.
Support Jingdong Cloud.
Support Meituan Cloud.
Improvements:
Improvement performance of cache sharing, decrease CPU and memory usage.
JuiceFS 4.2.3 (2017/11/15)¶
New Features：
Garbage collection: The client will scan all the objects to find out unused ones and clean them every week.
Add new attribute –gc to run garbage collection immediately.
Add P2P cache sharing for clients, they in the same intranet could share memory and disk cache to increase the hit ratio, improve the performance for hot data, decrease requests/cost of object storage.
JuiceFS 4.2.2 (2017/11/08)¶
New Features：
Support zero-copy concat for files, see juicefs merge -h for details. This feature require all the clients to upgrade to 4.2.1+, otherwise the merged files can’t be read by older clients
Allow mounting a non-empty directory.
Improvement：
Improve the background replication to make sure that writing will not be slowed down when there is no enough bandwidth for replicaiton.
Add DNS cache in process for UFile, improve the performace and stability.
Bug Fix:
Fixed the issue that cached root can’t be invalidated correctly.
JuiceFS 4.2.1 (2017/11/02)¶
New Features：
Improve the performance of random write (ten times faster).
Support Kubernetes Flex Volume, juicefs could be driver, implement init, mount and unmount
Support Microsoft Azure.
Support Netease Cloud.
Forbideen xattr as default, could enable by –enable-xattr
Open transfer encrypted between metadata service and object storage as default.
Auto synchronize replicated object storage, one time per week, don’t run juicefs heal manually.
Improvement：
Compact small chunks of single files into bigger one in background。
Improve read performance for small files, reduce the latency for first byte.
Reuse the HTTP connections for better performance and stability。
Increase the duration between retry。
Bug Fix：
Fixed the issue that juicefs overwrite the symlink during upgrading。
JuiceFS 4.1.26 (2017/10/5)¶
New Features：
Auto create bucket in object storage at you first mounting.
Support non-root user mount
Improvement：
Use musl to replace glibc for static linking.
Decrease memory usage on random write.
Optimize Copy operation of the UFile, improve the performance and decrease the bandwidth requirement.
juicefs auto selects Python2 or Python3 environment, support almost Linux distribution.
JuiceFS 4.1.25 (2017/9/16)¶
New Features：
juicefs version to show client version.
Add limit for the client currency amount, 10 for free plan, 100 for stanard trial, 1000 fro standard, no limit in enterprise plan.
Support Tencent Cloud COS(V5 API).
Add slow operation log for object storage(> 10s).
Improvement：
Exit with non-zero if there is something wrong in juicefs.
Increase the parallism of upload to 50 to speed up performance for write.
Cleanup cached chunk in local disk, when there is no enough space.
JuiceFS 4.1.24 (2017/9/3)¶
New Features
Change the nlink of directory to the number of inodes.
Change the quota from number of files to number of inodes. Each directory or symlink will also be billed as 256K.
Add operation time into oplog, add a syslog for operations which took longer than 10 seconds.
Make all the operations to be interruptable, which could allow the application could be killed or interrupted if metadata service or objct storage is not available.
Bug Fix：
Fix bug for the client aborting on write failure
JuiceFS 4.1.23 (2017/8/4)¶
Improvement：
Add local cache for uncompressed data block, highly improved random reading performance
Increase re-try times 3 to 10 for object storage operation
Exit after mount is successful
JuiceFS 4.1.21 (2017/7/17)¶
Improvement：
Optimize encrypted transportation.
Speed up the download in China.
JuiceFS 4.1.20 (2017/7/14)¶
Improvement：
Improve the performance on write with local cache.
JuiceFS 4.1.18 (2017/7/3)¶
Improvement：
Improve the performance of metadata service, support 300K+ QPS.
JuiceFS 4.1.17 (2017/6/20)¶
New Features：
Support Qiniu cloud storage.
Bug Fix：
Fix import file bug.
JuiceFS 4.1.15 (2017/6/2)¶
New Features：
Support Unix permissions based on username rather than local uid/gid. The clients tranlates local uid/gid to unique uid/gid based on username, make sure the users from different machines with same name works seamlessly like in same machines.
Improvement：
Support partitioning of objects to avoid the limitation in S3 that single partition can have limited qps. This should be specified when creating the filesystem, can’t be changed anymore.
Improve the performance to 10K+ QPS on metadata writing
Bug Fix：
Fixed the problem that causes wrong mtime on files after untar.
JuiceFS 4.1.13 (2017/5/24)¶
New Features：
Support KSYun
Support fast import from Object Storage, no data copy, more than 10K objects per second, see juicefs import -h
Improvement：
Cache the directories in client to enable faster lookup, which are tracked by metadata service and invalidate when needed. This is enabled by –dircache. The cached items expire after 5 minutes.
Don’t output log than quit when syslog is unavailable.
Decrease size of the client from 18MB to 4MB, download faster.
Improve the performance that copy to another object storage, reduce its impact on write performance.
Bug Fix：
Fix aborting issue when client can’t connect to the metadata service.
JuiceFS 4.1.11 (2017/4/26)¶
New Features：
Replicate data to another object storage in different region or cloud.
Check and sychronize all the objects in the replicated two object stores (juicefs heal）.
Change the size of directory to billing size rather than the logical sum of length of all files. nlink of directory means the number of files in that directory.
Bug Fix：
Fixed an overflow issue that causes the client to hang under heavy concurrent write workload.