Recently, while working with cloud providers and agent teams to deploy sandboxes, we found that beyond getting data into the sandbox, we also need to consider how the data required by tasks and the data generated during execution can be continuously used across tasks and stages. Sandboxes can be quickly created and destroyed with tasks, but data often needs to be retained, shared, and transferred.
As sandbox concurrency scales further, the challenges go beyond data persistence and shared access. Operational issues such as client scale, cache reuse, small files, and fault isolation become more prominent.
This article combines our practice in agentic reinforcement learning (RL) scenarios and JuiceFS deployment experience in cloud provider sandboxes to introduce how sandbox data is organized, accessed across clouds, and mounted. We’ll also discuss the problems and optimization directions encountered under large-scale concurrency.
How to manage long-lived data in short-lived sandboxes
Take agentic RL as an example. A task typically goes through task distribution, sandbox execution, trajectory collection, reward evaluation, and training optimization. Data generated during task execution continues to be used by subsequent stages.
First, the compute lifecycle and data lifecycle separate. When an agent task starts, it needs to read datasets, code repositories, dependency environments, and model weights. During execution, it continuously produces trajectories, logs, intermediate files, and evaluation results. The sandbox can be destroyed after the task ends, but some of this data still needs to be retained for replay, analysis, evaluation, and subsequent training.
Second, data sharing and task isolation must coexist. Each sandbox typically needs an independent working directory and access boundary, but different agents, evaluators, or training pipelines may access the same dataset, code repository, or reuse results from previous tasks.
Third, data needs to flow across different compute environments. When tasks are distributed across self-built environments and sandboxes from different cloud providers, maintaining independent data copies in each environment not only increases synchronization and management costs but also limits the flexibility of task scheduling.
How JuiceFS manages and accesses sandbox data
To address these data management issues, we introduce how JuiceFS organizes and accesses data in actual sandbox environments.
Organizing shared data and task data with directories
Different customers adopt different data organization methods based on their application characteristics. Based on the usage patterns of existing customers, a typical approach is:
- Datasets directory: stores core data such as evaluation datasets, training datasets, and benchmark task inputs.
- Repo directory: stores code repositories, toolkits, and snapshots of related dependency environments.
- Task directories: based on different tasks, runtime IDs, and training versions, independent directories are created in the file system to store data generated during each task run.
In this way, public data such as datasets and repos can be shared and accessed by multiple tasks, while each task has its own independent data space. At the same time, enterprises can decide which data needs to be retained long-term and which can be cleaned up after task completion, enabling more flexible data lifecycle management.
How cross-cloud sandboxes access the same data
Some customers use sandbox services from different providers across multiple cloud platforms simultaneously for reasons such as security, load balancing, and resource scheduling.
Our customers' core data (such as training sets and inference data) is usually stored in their own data platform or file system, while sandbox environments from different cloud providers run on their respective cloud infrastructures. The challenge is how sandboxes on different clouds can access the same data without maintaining separate data copies.
In such scenarios, JuiceFS Enterprise Edition's cross-cloud data access capability can be used to keep unified data on the customer's own platform while allowing sandboxes in different cloud environments to access it on demand. With fine-grained permission control, data isolation can be achieved between sandboxes, and resources such as datasets, code repositories, and evaluation data on different clouds can be managed within a unified namespace.
For example, after data is written from cloud A, a sandbox in cloud B can quickly read it and continue the subsequent task flow. This improves the flexibility of task scheduling and reduces the complexity of data management in multi-cloud environments.
JuiceFS data mounting mode for strongly isolated sandboxes
To enable sandboxes to actually access this data, we also need to solve the problem of how to mount the file system into isolated environments.
Currently, sandbox solutions provided by cloud providers are different somewhat from self-built sandboxes in enterprises. From the existing cloud sandbox architectures, most are based on lightweight virtualization technologies such as Firecracker, with each cloud provider optimizing to different degrees.
The core characteristic of such sandboxes is that they create a strongly isolated environment inside a virtual machine. This keeps it isolated from the host and other sandbox instances. Therefore, under this strong isolation architecture, data access cannot simply rely on host shared directories.
To address this, the current approach is to run an independent JuiceFS client alongside each sandbox. The client is launched with the sandbox through a sidecar-like mechanism provided by the cloud provider and exposes a local mount path to the application sandbox.
In this mode, each sandbox connects to the backend JuiceFS service with an independent identity, accesses the metadata service deployed in the customer environment through the VPC network, reads data from the corresponding object storage, and mounts shared or exclusive directories as needed by the task. For data that needs to be shared across multiple tasks and components, a shared directory enables unified access, while data unique to each task is isolated and protected through independent subdirectories, ensuring data security.
For permission management, the token access control capability provided by JuiceFS Enterprise Edition can further enable fine-grained permission management, ensuring secure and controllable data access between different sandboxes.
JuiceFS in cloud provider sandboxes
We’ve implemented JuiceFS in sandbox environments on Tencent Cloud and Alibaba Cloud.
Tencent Cloud: data mounting methods in agent runtime
Tencent Cloud uses the agent runtime mechanism, with three levels of isolation: file system level, instance level, and storage level.
- The first level is mounting block storage inside the sandbox. The block storage is exclusively used by a single sandbox, and the sandbox creates an independent file system on it. In this case, other environments cannot access this file system, achieving strong data isolation.
- The second level is mounting storage services provided by Tencent Cloud itself. Through mount, the corresponding storage can be mounted into a specific sandbox environment, which is essentially mounting inside the sandbox.
- The third level is the JuiceFS file system or subdirectory-level isolation. JuiceFS can mount the entire file system to a sandbox or only a specified subdirectory (subpath). Multiple sandboxes can share the same JuiceFS file system while dividing their data access scopes through different subdirectories, striking a balance between data sharing and task isolation.
Alibaba Cloud: FC and ACS
Alibaba Cloud has two product lines:
- The Function Compute (FC) Sandbox: Its implementation is similar to Tencent Cloud, also using a sidecar-like process mode to start a client in each sandbox to complete JuiceFS mounting.
- The Container Compute Service (ACS): It also provides sandbox services but with a different implementation. ACS integrates JuiceFS mounting capability into its driver component. In actual use, customers decide whether and how to use this capability. The related code is also planned to be open-sourced in its components.
Currently, whether on Tencent Cloud or Alibaba Cloud, JuiceFS mounting capability supports both Community Edition and Enterprise Edition.
Sandbox lifecycle management and mount control
In the current sidecar-like client mode, JuiceFS mounting is completed during sandbox startup, and the entire mount lifecycle is managed by the sandbox platform. For example, in Alibaba Cloud, the FC management platform is responsible; in Tencent Cloud, the agent runtime platform takes on this role.
Each sandbox corresponds to an independent JuiceFS client, which is created with the sandbox and completes directory mounting. The token, mount parameters, cache directory, and other configurations required during mounting are uniformly injected and managed by the runtime platform.
Under this mechanism, there are some differences between the Enterprise Edition and the Community Edition. JuiceFS Enterprise Edition provides more comprehensive management through the console and token capabilities, and cloud provider platforms have largely integrated these capabilities into their own mount management processes.
For the application side, the sandbox only needs to see the mounted data directory and does not need to directly manage the JuiceFS client. The creation, configuration, mounting, and exit of the client are all handled by the sandbox platform.
Challenges and optimization in large-scale agent sandbox scenarios
Client scaling bottleneck
Currently, one sandbox corresponds to one independent JuiceFS client, and each client is considered an independent client instance in the Enterprise Edition. As the number of sandboxes grows, the number of clients and the corresponding processes, connections, and metadata requests also increase.
However, most cloud provider sandbox implementations are based on Firecracker virtualization, and native Firecracker does not support passthrough mounting like VirtIO-FS. The number of clients a single file system can support is around 100,000. We’ve already received requests from some customers for millions or even more sandboxes. This makes client architecture optimization an important direction for future evolution.
One direction being explored is introducing a Mount Pod-like mode, where a JuiceFS client runs on the host or a designated node, and multiple sandboxes share the mounting capability. This can reduce the number of client instances and resource consumption, and further increase the supported sandbox scale.
Cache reuse and data loading efficiency
Currently, one sandbox corresponds to one independent JuiceFS client, and sandboxes may be distributed across different nodes. This makes it difficult to fully reuse client cache across tasks. Without a shared cache layer, a large amount of repeated data reads and loading pressure is directly transmitted to object storage; for cross-cloud sandbox scenarios, it may further increase bandwidth pressure on cross-cloud links.
To address this, we plan to combine task scheduling, host disk resources, and JuiceFS caching capabilities to improve cache reuse efficiency for hot data. For example, schedule related tasks to nearby nodes as much as possible, and use host disks to build a distributed cache, bringing frequently accessed data such as model datasets as close to compute as possible and reducing repeated loading from object storage.
When using cloud provider sandboxes, it’s necessary to combine the cache resources the cloud provider can offer; for self-built sandboxes, planning can be based on the customer's own infrastructure to balance performance, cache capacity, and resource cost.
Small files and metadata performance
During agent execution, data such as logs and trajectories often manifests as continuous small I/O writes, large numbers of small files, and frequent flushes. This places higher demands on JuiceFS metadata and write performance.
For such scenarios, we’ll work with application teams to explore optimization strategies, such as evaluating whether large numbers of small I/Os can be merged into larger-grained write requests, or using batch writes to reduce frequent small file operations. Adjusting the data write pattern on the application side can also help reduce pressure on the underlying metadata service.
Fault isolation and platform governance
In large-scale concurrency scenarios, it’s necessary to prevent a single abnormal task from affecting shared caches, metadata services, or object storage. For example, when abnormal access generates a large number of requests, its impact scope must be controlled as much as possible to avoid affecting other tasks.
If shared caches, shared clients, or client pools are further introduced, the original isolation boundaries will also change. After multiple sandboxes share underlying resources, fault isolation, resource limits, access permissions, and lifecycle management of shared clients need to be reconsidered.
On the other hand, as more platforms treat sandboxes as a public service for different teams or customers, data management will further extend to multi-tenant scenarios. Clear data and permission boundaries must be maintained between tenants, and different types of data require corresponding lifecycle policies.
Observability also needs to be improved accordingly, not only to observe client and cache status but also to understand overall storage usage, the running status of shared resources, and the scope of abnormal impact. This will provide a basis for subsequent resource governance.
From current practice, sandboxes are not difficult to create and destroy quickly; the real challenge is that the data behind them cannot disappear with them. Trajectories, logs, evaluation results, and training samples still need to be used by subsequent tasks, which makes storage gradually evolve from "providing a directory for the sandbox" to participating in the entire agent data pipeline over the long term.
As the scale grows from hundreds of thousands to millions, designs that were once simple and straightforward, such as sandbox clients and independent caches, also need to be reconsidered. We are still validating different approaches for client reuse, caching, and isolation, and we welcome exchanges with teams working on agent sandboxes, agentic RL, or related infrastructure about the problems they encounter.
If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.
