Scaling Flash Media Servers infrastructure

Posted on by

This post is also available in: Russian

Scalable online multimedia projects often require much more than just several video delivery servers. In this article we will cover the process of creating flexible infrastructure making you able to scale up to any number of Flash Media Servers. Using standard network level hardware load balancers often is not an option for RTMP/RTMPE delivery. Network level LB could bring extra restriction on available functionality and be very expensive. That’s why it is very usefull to make dedicated software load balancer to bring extra the flexibility without big expenditures.

We tested and optimized this approach in many deployments. In our scheme software load balancing is performed on Adobe Flash Media Servers (from one to many). It redirects visitors video requests to the most appropriate Flash Media Server. You could construct even geographically distributed cluster of Flash Media Servers with ease and flexibility of scaling.

Solution consist of:

  • TCP Load balancer
  • Flash Media Load Balancer
  • Edge FMS
  • Origin FMS
  • File Storage

Logical scheme of solution is shown below in Edge/Origin deployment:

When one Flash Media load balancer is not enough (about 1000 requests per second) you could place extra servers next to it and distribute load among them. It is possible because load balancer uses simple RTMP Redirect functionality with almost instant response.
Suitable TCP load balancing techniques:

  • Hardware TCP Load Balancers (Cisco, Juniper, 3COM)
  • Software load balancer (Haproxy)
  • DNS Round-Robin balancer

The most reliable with bigger performance solution is hardware load balancing. Additionally it could protect your project from DoS/DDoS attacks. But software load balancer like Haproxy could be a quickly deployed solution which could be launched on almost any Linux server.

Flash video player sends connect request to Flash Media Load Balancer or virtual IP of distributed load balancer. This request is initially received by TCP load balancer, it decides which Flash Media Load Balancer should process this request and forwards request to it (using techniques described above). Flash Media Load Balancer receives connect request and based on collected statistics from Flash Media Servers, visitor geography, availability of content in cache sends RTMP Redirect with actual URL of Flash Media Server (Edge) which is the best suitable for delivery of the content. Flash video player receives RTMP Redirect and connects to provided Edge server URL.

We propose to use following metrics from Flash Media Servers for load balancing decisions:

  • Active connections
  • Content in cache
  • Reponse time
  • ICMP check, TCP Handshake
  • Process queue

Flash Media Load Balancer could be developed on Flash Media Interactive Server 4 (SSAS, Access/Auth plugin). It collects load data from all Flash Media Servers delivering content. Intellectual balancing could give you best user experience and significantly decrease probability of delivery errors.

It should be mentioned that this Cache Aware load balancing could bring up to 30% increase in performance on the same hardware. This is because serving vide from the Edge cache is much faster (and you could place SSD drives in Edge servers for better performance). As example you could have exclusive Ad video campaign when almost all visitors of your portal watch this video clip. In case of standard load distribution all Flash Media Servers will have that clip in cache and it will influence on original video content performance on your site.

Flash Media Balancer features we developed in our deployments:

  • Load balancing among Flash Media Servers:
    • by server load
    • by content availability in cache
    • by geography
  • FMS Health check
  • Intellectual (many parameters) FMS monitoring
  • CDN integration (for some part of video library)
  • Performance Statistics

Often you need to limit access to the content based on visitor geography (as example when you have rights only on particular territory). In this case you should put auth plugin on every Flash Media Server (not on load balancer) acting as Edge. This auth plugin connects to ip geofiltering web service to determine location and apply restrictions.

This architecture was developed to make it simpler to integrate functionality in almost any multimedia project.

PS
Similiar algorithms are used in CDN networks to optimize performance. We hope they could be usefull for you also.

Leave a Reply