Analysis of VNF Performance and price

Price versus amount of resources

In this section, we analyze the prices of Amazon EC2 instances versus their size in terms of vCPU and memory. We then study the performance of different types of VNF instances using such instances in terms of a number of critical metrics: packet arrival rate, CPU utilization and packet loss rate.

The cost of operating Amazon EC2 instances (Amazon (2017)) is mainly reliant on their configuration  , i.e. the amount of virtual resources (ı.e. vCPU and memory) allocated to run these instances.

According to Amazon EC2 pricing list (Amazon (2017)), the lowest instance configuration is t2.micro, which operates with the least resources in terms of CPU and memory (i.e. 1 vCPU core and 1 GiB of memory). The price to run a single instance of t2.micro, for example, is $0.012 per hour whereas operating an instance of t2.small type, which is allocated a single vCPU and 2GiB RAM, is almost double the hourly rate at $0.023. This also means that having two t2.micro instances would cost a tenant $0.024 per hour. However, the tenant will have twice the processing capacity of a single t2.small. In other words, two t2.micro instances would in theory provide better performance than a single t2.small instance with merely the same price.

To better analyze whether this holds true for other types of EC2 instances, we evaluate how many t2.micro instances it would take to provide the same amount of resources of larger EC2 instance types, and how they are compared in terms of price.

Amount of resources versus VNF performance

In the previous subsection, we discussed VM prices according to Amazon EC2 (Amazon (2017)) compared to the amount of resources. In this section, we evaluate the performance of three VNFs namely a firewall, an IDS, and a NAT in this subsection. We ran Shorewall as a firewall (Shorewall (2017)), Snort as a network IDS (Snort (2017)), and the Ubuntu-based NAT.

Experimental Setup
We start the evaluation by generating UDP traffic between a pair of instances, where the VNF is installed inline to face the incoming traffic, process it, and then forward it to the destination. We ran the same experiment to examine the processing capacity of all VNFs. We generated traffic using iperf (iPerf (2017)), with this traffic increasing linearly with time. The purpose of increasing traffic load is to measure the CPU, memory, and disk utilization as well as packet loss for a VNF instance for different packet arrival rates. By doing this, we determine the highest packet processing capacity that each VNF can process per second in a particular instance.

Micro instance processing capacity
As previously highlighted, the t2.micro instance offers the most cost-effective access to VNF-hosting cloud instances. Thus, we look at how capable such instance is at hosting VNFs. For each VNF, we measure CPU, memory and disk utilization as well as packet loss as the received traffic increases over time.

Amount of resources versus performance
To compare the performance of instances of different sizes, we ran the same experiments for different instance types. For each of them, we identify the peak packet processing capacity (packets per second) as the maximum packet processing rate of the VNF when the CPU utilization reaches 90% or the packet loss reaches 10%, whichever first.

Discussion
Abstracting from the above observations, we extract the following outcomes.

– Input packet rate is crucial; CPU is the bottleneck
The incoming packet rate is the determining factor in the performance of VNFs. Consequently, anticipating such packet rate and its variance is a key element of dimensioning VNF embedding across any network

– Not all VNFs are the same
Every deployment of a VNF has its own performance ‘breaking point’, which varies across VNF types and we conjecture that it would vary also between different implementations of the same VNF. We observed that a low-demand VNF such as an IDS is easily hosted on a very modest cloud instance such as the EC2 t2.micro. However, to obtain similar packet rate capacity for a slightly more demanding VNF, such as a firewall, it requires hosting on a higher specification instance (in this case, t2.medium).

We can hence conclude that the type of the VNF and its implementation (in other words, the software running on the instance) will have a major impact on the packet processing capacity of the VNF. It is therefore a major challenge to write customized code that can optimize the VNF operations and further leverage the resources available in the instance where the network function is running.

– Instance specification is a loose indicator of performance
This observation has already been made by different research groups using both generic benchmarks and specific application profiles. This has been a motivator for our study, and indeed we have verified that such variance does exist even for NFV workloads. We found also the performance is not proportional to the amount of CPU and memory allocated for the VNF. This makes selecting the right instance based on the amount of the instance resources not accurate enough. As a result, the instance specification is only a loose indicator of its performance. It is therefore of utmost importance to test in advance the VNF when running on a particular instance, and evaluate its packet processing capacity as it may vary depending on the instance type.