{"id":1067,"date":"2010-12-21T15:28:34","date_gmt":"2010-12-21T12:28:34","guid":{"rendered":"http:\/\/blog.denivip.ru\/?p=1067"},"modified":"2013-08-05T14:12:37","modified_gmt":"2013-08-05T10:12:37","slug":"http-dynamic-streaming-configuring-web-server-cluster","status":"publish","type":"post","link":"http:\/\/blog.denivip.ru\/index.php\/2010\/12\/http-dynamic-streaming-configuring-web-server-cluster\/?lang=en","title":{"rendered":"HTTP Dynamic Streaming: Configuring Web Server Cluster"},"content":{"rendered":"<p style=\"text-align: center;\"><a href=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/Nanoscience_High-Performance_Computing_Facility.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-944    aligncenter\" title=\"High Perfomance\" src=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/Nanoscience_High-Performance_Computing_Facility-300x198.jpg\" alt=\"\" width=\"300\" height=\"198\" srcset=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/Nanoscience_High-Performance_Computing_Facility-300x198.jpg 300w, http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/Nanoscience_High-Performance_Computing_Facility-1024x678.jpg 1024w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Video content is now ubiquitous on the Web, but sometimes user experience leaves much to be desired. We have already <a href=\"http:\/\/blog.denivip.ru\/index.php\/2010\/11\/fms_highload\/?lang=en\">discussed<\/a> here enabling high performance configurations to deliver video content based on Adobe Flash Media Server. Still, sometimes HTTP delivery is more efficient. In this article, we would like to give you an overview of how to build a high performance video delivery system based on NGINX and Apache Web servers.<!--more--><\/p>\n<p>Video delivery based on HTTP Dynamic Streaming offers a number of benefits to the end user:<\/p>\n<ul>\n<li>It provides higher performance of FLASH applications compared to RTMP due to lower RAM utilization (RTMP video stream is buffered to RAM, while HTTP stream is buffered to HDD).<\/li>\n<li>Narrow-band users can pause video and wait until it downloads completely to a PC. They can enjoy the highest quality at any connection rate. This is of primary importance to Russia, where by no means all providers have moved to modern links with high bandwidth.<\/li>\n<li>While navigating a video, the content buffer is not reset.<\/li>\n<\/ul>\n<p>In this article, we will focus on building a high performance HTTP video content delivery system. The solution offered has several logical levels: network storage (disk array), backend servers (video fragment management), and a caching front-end. Using NGINX server as a front-end ensures efficient content caching and hence improves the overall system performance. Most optimum here is implementing intelligent query balancer based on front-end cache, but this deserves a separate article.<\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/test2.png\"><\/a><a href=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/22.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-942\" title=\"22\" src=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/22-300x290.png\" alt=\"\" width=\"300\" height=\"290\" srcset=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/22-300x290.png 300w, http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/22.png 698w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>It should be noted that, although the offered architecture is tailored for content distribution using Adobe Flash Access 2.0, it is also efficient for flv and f4v container delivery.<\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/scheme.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-912  aligncenter\" title=\"scheme\" src=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/scheme-300x132.jpg\" alt=\"\" width=\"344\" height=\"151\" srcset=\"http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/scheme-300x132.jpg 300w, http:\/\/blog.denivip.ru\/wp-content\/uploads\/2010\/12\/scheme.jpg 950w\" sizes=\"(max-width: 344px) 100vw, 344px\" \/><\/a><\/p>\n<p id=\"internal-source-marker_0.3270204039290219\">\n<p><strong>Preliminaries<\/strong><br \/>\nTo build the cluster, a link between Apache ver. =>2.2.x (video fragment origin) and NGINX =>0.8.x. For this purpose, you will need the f4fhttp module, supplied with Adobe Flash Media Server.<br \/>\nAlso, for video packaging, protection and fragmenting you will need the Adobe f4packager utility distributed as part of Adobe Flash Media Server.<\/p>\n<p>You can download f4fhttp and f4packager from the <a href=\"http:\/\/www.adobe.com\/products\/httpdynamicstreaming\/\">Adobe Web site<\/a>. There you can also find the description of video packaging procedure.<\/p>\n<ul>\n<li>The FLV format, also known as Flash Video. This is a popular format used for Web based video delivery. Currently it is used by such major video hosting sites as YouTube and Google Video.<\/li>\n<li>The F4V format was developed by Adobe and optimized for Adobe FMS to deliver HD content (for instance, DVR video recording on FMS is saved to this container).<\/li>\n<li>The F4F format (MP4 fragment) was also developed by Adobe to store encrypted video content. This format mandates Adobe Flash Access 2.0. F4F is closely related to f4v\/flv as they use the same technology for file segmentation, enabling fragmented file output by the server. For instance, when the seek operation in Flash player is used, only the needed fragments are loaded from the Web server. For more details on format specification, please visit <a href=\"http:\/\/help.adobe.com\/en_US\/HTTPStreaming\/1.0\/Using\/WS4e5b2ae7575fffb8-4b53131d1272fa47d71-7ffc.html\">Adobe\u2019s site<\/a>.<\/li>\n<\/ul>\n<p>High performance storage is integral to streamlining the solution operation. For instance, with ZFS and hardware Flash accelerators Sun Unified Storage 7000, an optimum content output rate can be reached. Similar performance could be achieved with other vendors also.<\/p>\n<p><strong>Configuring Apache Backend Server<\/strong><br \/>\nIn our configuration, Apache will function as an Origin server, using f4fhttp module to output segments of *.f4f video files to front-end servers. This solution is not limited to f4f video streaming. For FLV delivery, instead of Apache you can use NGINX built with ngx_http_flv_module (downloadable from <a href=\"http:\/\/sysoev.ru\/nginx\/docs\/http\/ngx_http_flv_module.html\">module web page<\/a>). In this case, the backend NGINX server will output FLV segments, and front-end NGINX will cache them. This also minimizes delays, as NGINX will no longer have to download the whole file to output segments to the Web server.<\/p>\n<p>First of all, let\u2019s enable f4fhttp module loading at Apache startup. To do this, add the following string to httpd.conf file:<\/p>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">LoadModule f4fhttp_module modules\/mod_f4fhttp.so<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Before doing this, make sure the files of the module are available in this directory.<\/p>\n<p>Now let\u2019s create a directory for video files delivered to customers, and change its owner:<\/p>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/><\/div><\/td><td><div class=\"text codecolorer\">mkdir \/var\/www\/video<br \/>\nchown -c apache:apache \/var\/www\/video<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Having created the directory, add the following code to the Web server configuration file:<\/p>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/>9<br \/>10<br \/>11<br \/>12<br \/>13<br \/>14<br \/>15<br \/>16<br \/><\/div><\/td><td><div class=\"text codecolorer\">#Set the DocumentRoot directory<br \/>\nDocumentRoot &quot;\/var\/www\/video&quot;<br \/>\n#Define Web server options for the directory<br \/>\n&amp;lt;Directory \/&amp;gt;<br \/>\nOptions FollowSymLinks<br \/>\nAllowOverride None<br \/>\n&amp;lt;\/Directory&amp;gt;<br \/>\n#Set the parameters of operating with the directory<br \/>\n&amp;lt;Location \/vod&amp;gt;<br \/>\n#Enable content streaming within the directory<br \/>\nHttpStreamingEnabled true<br \/>\nHttpStreamingEnabled true<br \/>\n#Specify physical path to video containing folder<br \/>\n#(used with symlink)<br \/>\nHttpStreamingContentPath &quot;\/var\/www\/video\/vod&quot;<br \/>\n&amp;lt;\/Location&amp;gt;<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Now the backend server is configured, you can restart it and proceed to the next stage.<\/p>\n<p><strong>Configuring NGINX front-end server<\/strong><br \/>\nHaving installed NGINX, change the configuration file as follows:<\/p>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/><\/div><\/td><td><div class=\"text codecolorer\">http {<br \/>\n\u2026............<br \/>\nproxy_cache_path \u00a0\/data\/cache levels=1:2 keys_zone=my-cache:8m max_size=80G;<br \/>\n\u2026............<br \/>\n}<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>This defines the cache zone to store video fragment cache of 80GB.<\/p>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/>9<br \/>10<br \/>11<br \/>12<br \/>13<br \/>14<br \/>15<br \/>16<br \/><\/div><\/td><td><div class=\"text codecolorer\">location \/ {<br \/>\n\u2026.........<br \/>\n#Set the caching key. We are going to cache with parameters.<br \/>\n#As our file names are always<br \/>\n#unique, this guarantees that cache is also unique<br \/>\nfastcgi_cache_key &quot;$server_addr:$server_port$request_uri&quot;;<br \/>\n<br \/>\n#Set the address of earlier configured Apache backend server<br \/>\nproxy_pass http:\/\/10.0.0.1:80;<br \/>\n<br \/>\n#Set cache related parameters<br \/>\nproxy_cache my-cache;<br \/>\nproxy_cache_valid \u00a0200 302 \u00a060m;<br \/>\nproxy_cache_valid \u00a0404 \u00a0\u00a0\u00a0\u00a0 1m;<br \/>\n\u2026........<br \/>\n}<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>Having modified the configuration file, launch the NGINX Web server and try to load video using Flash player configured to HTTP Dynamic Streaming. If servers are linked correctly, video content streaming will begin. Cache structure will be generated in the \/data\/cache directory.<\/p>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">#gizmo&amp;gt; watch ls -lah \/data\/cache<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">&nbsp;Every &nbsp;2.0s: ls -lah \/data\/cache\/<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/><\/div><\/td><td><div class=\"text codecolorer\">Fri Dec 10 18:46:29 2010<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<div class=\"codecolorer-container text default\" style=\"overflow:auto;white-space:nowrap;width:540px;\"><table cellspacing=\"0\" cellpadding=\"0\"><tbody><tr><td class=\"line-numbers\"><div>1<br \/>2<br \/>3<br \/>4<br \/>5<br \/>6<br \/>7<br \/>8<br \/>9<br \/>10<br \/>11<br \/>12<br \/>13<br \/>14<br \/>15<br \/>16<br \/>17<br \/><\/div><\/td><td><div class=\"text codecolorer\">total 68K<br \/>\ndrwxr-xr-x 17 nginx nginx 4.0K Dec \u00a03 14:18 .<br \/>\ndrwxr-xr-x 10 root \u00a0root \u00a04.0K Dec \u00a04 18:33 ..<br \/>\ndrwx------ \u00a03 nginx nginx 4.0K Dec \u00a03 14:16 0<br \/>\ndrwx------ \u00a04 nginx nginx 4.0K Dec \u00a03 14:17 1<br \/>\ndrwx------ \u00a05 nginx nginx 4.0K Dec \u00a03 14:19 2<br \/>\ndrwx------ \u00a07 nginx nginx 4.0K Dec \u00a03 14:18 3<br \/>\ndrwx------ \u00a03 nginx nginx 4.0K Dec \u00a03 14:17 4<br \/>\ndrwx------ \u00a04 nginx nginx 4.0K Dec \u00a03 14:18 6<br \/>\ndrwx------ \u00a05 nginx nginx 4.0K Dec \u00a03 14:16 7<br \/>\ndrwx------ \u00a03 nginx nginx 4.0K Dec \u00a03 14:13 8<br \/>\ndrwx------ \u00a05 nginx nginx 4.0K Dec \u00a03 14:15 9<br \/>\ndrwx------ \u00a03 nginx nginx 4.0K Dec \u00a03 14:18 a<br \/>\ndrwx------ \u00a05 nginx nginx 4.0K Dec \u00a03 14:18 b<br \/>\ndrwx------ \u00a03 nginx nginx 4.0K Dec \u00a03 14:17 c<br \/>\ndrwx------ \u00a05 nginx nginx 4.0K Dec \u00a03 14:18 e<br \/>\ndrwx------ \u00a04 nginx nginx 4.0K Dec \u00a03 14:15 f<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n<p>For optimal performance of this configuration, you will also need to optimize OS kernel settings and NGINX caching preferences. Follow us for Linux OS kernel tuning and more.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Video content is now ubiquitous on the Web, but sometimes user experience leaves much to be desired. We have already discussed here enabling high performance configurations to deliver video content based on Adobe Flash Media Server. Still, sometimes HTTP delivery is more efficient. In this article, we would like to give you an overview of [&hellip;]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20,12],"tags":[228,205,38,14,13],"_links":{"self":[{"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/posts\/1067"}],"collection":[{"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/comments?post=1067"}],"version-history":[{"count":5,"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/posts\/1067\/revisions"}],"predecessor-version":[{"id":4875,"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/posts\/1067\/revisions\/4875"}],"wp:attachment":[{"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/media?parent=1067"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/categories?post=1067"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.denivip.ru\/index.php\/wp-json\/wp\/v2\/tags?post=1067"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}