«Abstract. A ﬁlesystem-level storage cloud offers network-ﬁlesystem access to multiple customers at low cost over the Internet. In this paper, we ...»
A Comparison of Secure Multi-tenancy Architectures
for Filesystem Storage Clouds
Anil Kurmus1, Moitrayee Gupta 2, Roman Pletka1, Christian Cachin1, and Robert
IBM Research - Zurich
Department of Computer Science and Engineering, UCSD
Abstract. A ﬁlesystem-level storage cloud offers network-ﬁlesystem access to
multiple customers at low cost over the Internet. In this paper, we investigate
two alternative architectures for achieving multi-tenancy securely and efﬁciently in such storage cloud services. They isolate customers in virtual machines at the hypervisor level and through mandatory access-control checks in one shared operating-system kernel, respectively. We compare and discuss the practical secu- rity guarantees of these architectures. We have implemented both approaches and compare them using performance measurements we obtained.
1 Introduction Storage cloud services allow the sharing of storage infrastructure among multiple cus- tomers and hence signiﬁcantly reduce costs. Typically, such services provide object or ﬁlesystem access over a network to the shared distributed infrastructure. To support multiple customers or tenants concurrently, the network-ﬁlesystem-access services must be properly isolated with minimal performance impact.
We consider here a ﬁlesystem storage cloud as a public cloud storage service used by customers to mount their own ﬁlesystems remotely through well-established network ﬁlesystem protocols such as NFS and the Common Internet Filesystem (CIFS, also known as SMB). Such a service constitutes a highly scalable, performant, and reliable enterprise network-attached storage (NAS) accessible over the Internet that provides services to multiple tenants.
In general, a cloud service can be run at any of the following increasing levels of
– Hardware level: server hardware, OS, and application dedicated per client.
– Hypervisor level: share server hardware, and use virtualization to host dedicated OS and application per client.
– OS level: share server hardware and OS, and run a dedicated application per client.
– Application level: share server hardware, OS, and application server among clients.
Work done at IBM Research - Zurich.
2 Anil Kurmus, Moitrayee Gupta, Roman Pletka, Christian Cachin, and Robert Haas Intuitively, the higher the level of multi-tenancy, the easier it seems to achieve a resource- efﬁcient design and implementation; at the same time, though, it gets harder (conceptually and in terms of development effort) to securely isolate the clients from each other.
In this paper, we investigate a hypervisor-level and an OS-level multi-tenant ﬁlesystem storage cloud architecture, and compare them in terms of performance and security.
The hypervisor-level multi-tenancy approach is based on hardware virtualization (with para-virtualized drivers for improved networking performance). We refer to this architecture as the virtualization-based multi-tenancy (VMT) architecture. The OS-level multi-tenancy approach uses mandatory access control (MAC) in the Linux kernel and is capable of isolating customer-dedicated user-space services on the same OS. Such an architecture may also leverage, for instance, OS-level virtualization technologies such as OpenVZ or Linux Containers (LXC). We refer to this architecture as the operatingsystem-based multi-tenancy (OSMT) architecture in the remainder of this paper.
We have implemented both approaches on real hardware in the IBM Scale-out NAS (SONAS)  and the IBM General Parallel Filesystem (GPFS)  technologies. We used open-source components such as KVM  with virtio networking for virtualization and SELinux (http://selinuxproject.org/) for MAC.
Section 3 describes the architecture of a ﬁlesystem storage cloud and introduces the two designs. Section 4 deﬁnes an adversary model and discusses the security of both architectures according to this model. Section 5 presents the implementation and benchmark results. Related work is discussed in Section 6.
2 Background One can distinguish the following categories of general-purpose storage clouds (ignoring
storage clouds that provide database-like structures on content):
– Block storage clouds, with a block-level interface, i.e., an interface that allows the writing and reading of ﬁxed-sized blocks. Examples of such clouds include Amazon EBS.
– Object storage clouds, composed of buckets (or containers) that contain objects (or blobs). These objects are referred to by a key (or name). The API is usually very simple: typically a REST API with create and remove operations on buckets and put, get, delete, and list operations on objects. Example of such storage clouds include Amazon S3, Rackspace Cloudﬁles, and Azure Storage Blobs.
– Filesystem storage clouds, with a full-ﬂedged ﬁlesystem interface, therefore referred to also as “cloud NAS.” Examples of such clouds include Nirvanix CloudNAS, Azure Drive, and IBM Scale-Out Network Attached Storage (SONAS).
Application-level multi-tenancy is sometimes also referred to as native multi-tenancy.
Some authors consider it the cleanest way to isolate multiple tenants . However, achieving multi-tenancy securely is very challenging and therefore not common for ﬁlesystem storage clouds. The reasons lie in the complex nature of this task: unlike other types of storage clouds, ﬁlesystem storage clouds possess complex APIs that have evolved over time, which leads to large attack surfaces. The vulnerability track record of these applications seems to conﬁrm this intuition. CIFS servers were vulnerable A Comparison of Secure Multi-tenancy Architectures for Filesystem Storage Clouds 3 to various buffer-overﬂows (e.g., CVE-2010-3069, CVE-2010-2063, CVE-2007-2446, CVE-2003-0085, CVE-2002-1318, see http://cve.mitre.org/), format string vulnerability leading to arbitrary code execution (CVE-2009-1886), directory traversals (CVE-2010-0926, CVE-2001-1162), while NFS servers were also vulnerable to similar classic vulnerabilities as well as more speciﬁc ones such as ﬁlehandle vulnerabilities .
Moreover, adding multi-tenancy support into these server applications would require signiﬁcant development (e.g., in order to distinguish between different authentication servers for speciﬁc ﬁlesystem exports) which will most likely result in new vulnerabilities.
We discuss in Sections 3 and 4 architectures with lower levels of multi-tenancy. They effectively restrict the impact of arbitrary code execution vulnerabilities to the realm of a single tenant: by deﬁnition, this cannot be achieved with application-level multi-tenancy.
This paper targets the IBM SONAS  platform, which evolved from the IBM ScaleOut File Services (SoFS) . IBM SONAS provides a highly scalable network-attached storage service, and therefore serves as a typical example of a ﬁlesystem storage cloud.
IBM SONAS currently contains support for hardware-level multi-tenancy according to the architectures discussed in this work. Adding a higher-level of multi-tenancy is an important step to reduce the cost of a cloud-service provider.
3 System Description
Section 3.1 gives an overview of the general architecture of a ﬁlesystem storage cloud.
Section 3.2 describes the MAC policies which are used in both architectures.
Sections 3.3 and 3.4 introduce the two alternatives, detailing the internals of the interface nodes, the key element of the ﬁlesystem storage cloud architecture.
3.1 General Description Figure 1 depicts the general architecture of a ﬁlesystem storage cloud that consists of the
Fig. 1. General architecture of a ﬁlesystem storage cloud.
4 Anil Kurmus, Moitrayee Gupta, Roman Pletka, Christian Cachin, and Robert Haas – Customers and users: A customer is an entity (e.g., a company) that uses at least one network ﬁle system. A customer can have multiple individual users. We assume that multiple customers connect to the ﬁlesystem storage cloud and that each customer has a separate set of users. Data is separated between users from distinct customers, and user IDs are in a separate namespace for each customer. Hence two distinct customers may allocate the same user ID without any conﬂict on the interface nodes or in the storage back-end.
– Interface nodes and cluster: An interface node is a system running ﬁler services such as NFS or CIFS daemons. Interface nodes administratively and physically belong to the cloud service provider and serve multiple customers. A customer connects to the ﬁlesystem storage cloud through the interface nodes and mounts its ﬁlesystems over the Internet. Multiple interface nodes together form an interface cluster, and one interface node may serve multiple customers. A customer connects only to nodes in one interface cluster.
– Shared back-end storage: The shared back-end storage provides block-level storage for user data. It is accessible from the interface clusters over a network using a distributed ﬁlesystem such as GPFS . It is designed to be reliable, highly available, and performant. We assume that no security mechanism exists within the distributed ﬁlesystem to authenticate and authorize nodes of the cluster internally.
– Customer boarding, unboarding, and conﬁguration: Typically, interface nodes must be created, conﬁgured, started, stopped, or removed when customers are boarded (added to the service) or unboarded (removed from the service). This is performed by administration nodes not shown here, which register customer accounts and conﬁgure ﬁlesystems. Ideally, boarding and unboarding should consume a minimal amount of system resources and time.
As an example, a customer registers a ﬁlesystem with a given size from the ﬁlesystem storage cloud provider, and then conﬁgures machines on the customer site that mount this ﬁlesystem. The users of the customer can then use the cloud ﬁlesystem similar to how they use a local ﬁlesystem. Customers connect to the interface cluster via a dedicated physical wide-area network link or via a dedicated VPN over the Internet, ideally with low latency. The cloud provider may limit the maximal bandwidth on a customer link.
To ensure high availability and high throughput, a customer accesses the storage cloud through the clustered interface nodes. Interface nodes have to perform synchronization tasks within their cluster and with the back-end storage, generating additional trafﬁc. An interface node has three network interfaces: one to the customer, one to other nodes in the cluster, and one to the back-end storage.
Dimensioning. The size of a ﬁlesystem storage cloud is determined by the following parameters, which are derived from service-level agreements and from the (expected or observed) load in the system: the number of customers c assigned to an interface cluster, the number of interface nodes n in a cluster (due to synchronization overhead, this incurs a trade-off between higher availability and better performance), and the number of clusters m attached to the same storage back-end.
Customer and user authentication. Describing customer authentication would exceed the scope of this work; in practice, it can be delegated to the customer’s VPN endpoint in the A Comparison of Secure Multi-tenancy Architectures for Filesystem Storage Clouds 5 premises of the service-cloud provider. The authentication of users from a given customer also requires that customers provide a directory service that will serve authentication requests made by users. Such a directory service can be physically located on the customer’s premises and under its administration or as separate service in the cloud. In either case, users authenticate to an interface node, which in turn relays such requests to the authentication service of the customer.
3.2 Mandatory Access Control Policies We use mandatory access control on the ﬁler services. In case of their compromise, MAC provides a ﬁrst layer of defense on both architectures. For practical reasons, we have used SELinux. Other popular choices include grsecurity RBAC  or TOMOYO. These MAC systems limit the privileges of the ﬁler services to those required, effectively creating a sandbox environment, by enforcing policies that are essentially a list of permitted operations (e.g., open certain ﬁles, bind certain ports, fork,... ).
As an example, the policies on the interface nodes basically permit the ﬁler services to perform the following operations: bind on their listening port and accept connections, perform all ﬁlesystem operations on the customer’s data directory (which resides on a distributed ﬁlesystem), append to the ﬁler log ﬁles, and read the relevant service conﬁguration ﬁles.
The protection provided by these policies can be defeated in two ways. One possibility is if the attacker manages to execute arbitrary code in kernel context (e.g., through a local kernel exploit), in which case it is trivial to disable any protections provided by the kernel, including MAC. The second possibility is by exploiting a hole in the SELinux policy, which would be the case, for example, if a ﬁler service were authorized to load a kernel module.
An important example of the beneﬁt of these policies is the restriction of accessible network interfaces to the customer and intra-cluster network interfaces only. Another example is the impossibility for processes running in the security context of the ﬁler services to write to ﬁles they can execute, or to use mmap() and mprotect() to get around this restriction. In practice, this means, for example, that an attacker making use of a remote exploit on a ﬁler service cannot just obtain a shell and download and execute a local kernel exploit: the attacker would have to ﬁnd a way to execute the latter exploit directly within the ﬁrst exploit, which, depending on the speciﬁcs of the vulnerabilities and memory protection features, can be impossible.