This post is also available in: Russian
In this article, we will discuss how you can use the FMS Access plugin for load balancing among all of its FMS fmscore processes. Also, we will brief you on how to apply the Access plugin to balance VoD content load over multiple FMS servers.
Access plugin is another level of FMS security, which allows you to perform lightweight authorization of incoming connections before they reach the level of SSAS applications in the fmsedge process.
The Access plugin provides the following features:
- Analyze the properties of a client connection
- Analyze server statistics
- Analyze server statistics
- Send requests to the external systems
Once a client connection arrives to the fmsedge process, fmsedge passes connection control to the Access plugin (if installed). The Access plugin business logic can now decide on how to handle this connection: accept, reject or forward it to another FMS.
Distributing FMS load among its fmscore processors
One of the most fascinating features offered by the Access plugin is the ability to make a dynamic selection of fmscore process to forward a connection, right at the moment when the inbound connection arrives to fmsedge. This helps to more or less flexibly manage client connections at the FMS level, according to your system architecture and your resources. For example, this way you can guarantee the quality of service to certain groups of customers.
The process of binding the connection to fmscore process is well defined in the Adobe documentation. Here we would like to give you an example of obtaining the fmscore process ID to which to attach a connection obtained from the client connection URI. It is clear that originally this identifier should be added to the URI, for instance, within the player used to play back the content. The player could retrieve the ID from the load balancer or calculate it based on the parameters of requested content. We will use the example later in the paper, when discussing load balancing with SSAS applications.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | void MyAdaptor::onAccess(IFCAccess* pAccess) { ......... switch(pAccess->getType()) { case IFCAccess::CONNECT: { // get client's URI const char* uri = pAccess->getValue("s-uri"); const char* tmp = 0; // marker where core id is in the URI const char *pName = "coreId="; if(uri && (tmp=strstr(uri, pName))!=0) { // get a fmscore id const char *coreId = tmp + strlen(pName); // set the fmscore id if(!pAccess->setValue("coreIdNum", coreId)) { // log that set is failed } } break; } } ............ } |
VoD content load balancing
As a good example of using the Access plugin, we will describe and present the pieces of code of the load balancing system implemented for the VoD content broadcast platform. Please note that this is just one of the possible options, still it is quite functional and is used in video portals with large audience. However, the efficiency of the architecture certainly depends on the content type.
Suppose we have a pool of Edge servers with the FMS server providing access to the VoD content using a standard SSAS VoD application. The Edge servers receive content from the Origin servers that have a direct access to the VoD content storage. This means that Edges should have proxying enabled, so they could keep part of the content in the local cache on the disс.
So, our goal is to develop a more or less intelligent load balancer for a video platform, which will reduce the load on this platform. Reducing the load assumes, for each Edge server, efficient utilization of its resources, such as cache, network interface bandwidth, CPU, etc.
Fig. 1. Load balancing for VoD content architecture
For this purpose, we have developed two SSAS applications and an Access plugin. The solution architecture is shown in Fig. 1. On the Edge server, we are going to install the Access plugin to attach the inbound client requests to the proper fmscore process. At the same location, let’s install the edge_app SSAS application, which will collect statistics related to the file cache, the number of active users, etc., by calling FMS AdminAPI. Another SSAS application, balancer_app, will be responsible for initial distribution of user requests to play back certain pieces of content. So, in fact, this is quite similar to our load balancer.
So, the client must initially access the Load Balancer, passing the content name to it. In response, it will get the Edge Server ID and fmscore process ID to which the Access plugin will attach the client request. The load balancer periodically collects statistics (file cache content , performance characteristics, etc.) from the Edge servers. Based on this statistics, it distributes the arriving client requests using the RTMP redirect method.
Edge_app SSAS application
We need this SSAS application to periodically collect statistics from FMS and its cache. The statistics is gathered by calling the FMS AdminAPI methods. For our purposes, we need only 2 methods: getFileCacheStats() and getServerStats(). The first method provides statistics on the file cache in RAM grouped by the fmscore processes. The second method provides the server statistics: the number of active connections, bandwidth allocation, CPU utilization, etc.
Also, the application configuration file can accommodate the physical server characteristics which the load balancer can factor in to distribute the queries: the number of fmscore processes, the maximum server throughput, etc.
All these characteristics and properties are processed and cached to be used by the balancer_app SSAS application. For example, this way you can dump statistics from the Edge server:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
'srvStats' => { 'appURI' => 'rtmp://192.168.1.10/vod', 'maxOutBandwidth' => 6250000, 'cpuUsage' => 0, 'memoryUsage' => 24, 'phisicalMem' => 35889152, 'bwIn' => 0, 'bwOut' => 0, 'connected' => 0 }, 'cacheStats' => { 'cache' => { 'cache' => { 'sample_1.flv' => [ { 'coreId' => 2, 'useCount' => 25 } ] }, 'cores' => { '0' => 0, '1' => 0, '2' => 1 } } } |
Balancer_app SSAS application
This application consolidates statistics from all Edge servers by polling edge_app SSAS applications. Also, it functions as the primary entry point for all client requests for content playback, i.e., based on the collected statistics it looks for a proper Edge server among the available servers.
For instance, you can enable load balancing based on just two criteria:
- whether the content is in the file cache
- the number of active clients
The proper Edge server is found as follows. If the needed content is present in the file cache, the least loaded fmscore process is found among all Edges where the content is present. Alternatively, if there is no content in the cache, then simply the least loaded Edge server and fmscore process are found.
After the Edge server and fmscore process are found, the application must perform RTMP redirect returning a response to the client application containing Edge server address and the ID of its fmscore process. It would look like this:
1 2 3 4 5 6 7 8 9 10 | application.onConnect = function(client, contentName) { ....... // find an edge server var edge = this.findEdge(contentName); // find core process id at it var coreId = this.findCoreId(edge); var errObj = {errorMsg: "Rejected due to redirection!", coreId: coreId}; application.redirectConnection(client, edge.srv.appURI, "Redirected!", errObj); ........... } |
Access plugin
The Access plugin must be installed on the Edge server to enable correct attachment of inbound connections based on an ID passed by the load balancer (SSAS balancer_app) to fmscore process. For additional information, please refer to the beginning of this Part or to the Adobe Website.
Сlient application (player)
The client application (a player or an air application) should be able to interact with the load balancer (SSAS balancer_app), i.e. to make a request, interpret the response with RTMP redirect and properly construct the URI with the fmscore process ID. See Fig. 2.
Fig. 2. Interaction of the client application and the load balancer
First, the client application establishes a connection to the load balancer and provides to it the name of the content that the customer wants to play back:
1 2 | var nc = new NetConnection(); nc.connect(balancer_app_uri, "sample_1.flv"); |
In response, the load balancer finds a suitable Edge server and fmscore process, and makes RTMP redirect (the command is sent to the user), specifying in the redirect parameters the Edge server URI and fmscore process ID. A client application, for example, can handle the response as follows:
1 2 3 4 5 6 7 8 9 10 11 | public function onNCStatus(event:NetStatusEvent) : void { ........... if(event.info.code=="NetConnection.Connect.Rejected) { var info:Object = event.info; var app:Object = info.application; if(info.ex && info.ex.code==302) { var coreId = app.coreId); } } ............. } |
Finally, the client application must connect to the Edge server, passing the URI with the received identifier of the fmscore process to the Edge server, to be processed by the Access plugin:
1 2 3 4 5 |
...... var nc = new NetConnection(); var edgeURI = info.ex.redirect+"?coreId="+app.coreId; nc.connect(edgeURI); ...... |
Summary
This is but one of the options to enable load balancing on a video platform, still it is quite flexible to approach the issue correctly. For instance, in order to extend this example to live content balancing, it is sufficient to accumulate slightly different statistics on the Edge servers and ignore the file cache. Moreover, for DVR content, the solution would not require any change at all.