This post is also available in: Russian

Launching of large-scale online video portals requires a great deal of precision and accuracy at the design stage. In addition to complex infrastructure of a Web platform (the portal itself), we have to deal with a no less complex infrastructure video platform (as content delivery is only part of the platform). In this post, we will discuss some of the important principles of online video service design that we have learned over the five years of working on similar projects.

A critical part of every video project is the video platform, so let’s discuss it first.

Content delivery On-site video viewing requires a good server capacity, as each user takes a bandwidth of several Mbps. One of the solutions to the problem is to deliver content via CDN service providers, although independent content delivery is a better choice if you need to ensure a higher cost-efficiency or CDN services cannot be used for some other reason. This requires installation of a video content delivery platform.

Edge servers are used for content delivery to end users. In recent years, the implementations based on HTTP have been highly popular (HTTP Dynamic Streaming, HTTP Live Streaming, HTTP Progressive Download). In case of HTTP DS/LS, the Edge server caches segments (chunks) of video content. For optimal use of computing power, user requests should be sent to a server where the requested content is available in the cache. Such technology is called "content aware load balancing". Another important point is that the Edge servers should be able to put all their content or at least all their hot content to the total cache (to avoid delivery from the data storage). Therefore, knowing the size of your video content library, you can calculate the total effective disk space on your edge servers. Also, from requirements for peak number of simultaneous viewers you can calculate the total bandwidth you need to ensure from the edge servers to the open Internet. Also it is worth noting that the efficiency of making sites "closer" to the user is not always as obvious as it seems. If you decide to independently deliver video to users, you’d probably find our load balancer useful both as a server software to run on your hardware, and as a cloud service.

Content protection When handling premium content, it is essential to ensure protection of content. Typically, this is achieved using DRM solutions (e.g., Adobe Flash Access). As this module is highly critical, it is important to install it behind your DDoS protection tools (e.g., IPS). It is also important to consider license caching on the user side. A large number of video views generates a large number of requests to the license server. To minimize the load, you should set the allowed level of license caching (it is usually specified by the copyright holders). Recommendations of the next section apply to implementing of statistics collection at the license server.

Statistics Large-scale online video services generate tremendous volumes of user action statistics. This data is required to calculate financial results and implement user-friendly features (such as resuming playback from the last position, viewing recommendations), etc. To ensure that your service retains high performance under peak load and easily accommodates your audience growth, you should provide for an adequate architecture at the design phase. Statistics collection includes the following steps that require high quality of design:

Collection of raw data – information about video views (in case of HTTP DS – chunk download) can be aggregated by logs of output (edge) servers. If you plan to use CDN provider services, you should check the available CDN logs and verify the guarantees of their accuracy and completeness. As an alternative, you can use a statistics collection service. By the way, we have such service and it is called Video Analytics. It receives events from video players and stores them in formats allowing for quick access. Regular log files, cache storage and other technologies allowing to collect enormous data volumes without significant spending on hardware capacity are also critical to the first stage.

Initial processing Depending on what kinds of output data is needed, you should duly prepare the data. At this stage, it is better to migrate data from intermediate high-speed storages to a relational database for ease of processing.

Preparation of aggregated data Often statistics is collected to calculate financial parameters (e.g., for revenue sharing purposes). In this case, the data is prepared over long periods, including selection from a huge number of records with very complex relations.

Data archiving It is very important to use dedicated servers to calculate statistics, as over time the amount of information may exceed reasonable limits. For example, 10 million views of video content per day result in a yearly traffic of 3.6 billion views (a huge amount, if you need to store detailed viewing history). Any database, even a very powerful one, would get congested having to select from such number of entries. It is therefore pretty important, at a certain stage, to migrate the outdated and very rarely used data to separate servers. An alternative is to use database sharding by record dates.

Finally, here is a useful presentation from our Yahoo! colleagues sharing their experience on building a video platform for their services: Here is an interesting fact from the presentation: Yahoo! is using Rhozet and ffmpeg, which is not surprising, since you can barely rely on ffmpeg alone. For instance, in our projects we usually create content preparation (transcoding) farms based on Rhozet Carbon.

Another important part of video service is a web platform. Each project has its own Web platform specifics, so we’d just outline some of the key points that are worth considering during the video service design phase. It is also worth keeping in mind the common guidelines of developing of high-performance software. In the case of video services, most of all the architecture uses to be affected by the following requirements:

• It supports a variety of interfaces and platforms and can service various devices using a single kernel and a set of custom implementations (PC, iPad, iPhone, Android, TV)
• High degree of personalization – recommendations, viewing history, social profiles – create a tendency towards creating a unique portal and interface for each user.
• High interactivity – the users are sensitive to video service response rate: Fast page change, rapid change of video content in your player

Front End As Front End servers, lightweight Web servers such as NGINX and lighttpd, are used. At the design phase, it is important to understand how Web page caching is going to run. The choice of technology depends on many factors:

• How dynamic the Web pages are: the higher the personalization, the lower the gain of caching whole pages and more difficult to implement personalization in the future (e.g., the service that provides recommendations based on preferences of a specific user )
• How difficult it is to prepare pages (calculation duration)
• How much information will be delivered by AJAX

Depending on the specifics of your project, we recommend you to use different mechanisms for caching (caching of full pages, blocks or data only). Here you can find details on testing the performance of PHP frameworks, template engines and NoSQL databases.

Protection against computer bots and spammers (such as the Captcha tool) should be deployed on separate servers. They require significant computing power and can cause congestion and failure of basic services. You can find a nice selection of Yahoo!’s recommendations on how to optimize your website here.

Cache Servers As a cache server, MemCached is often used. Due to absence of built in replication and presence of some nice features, many projects start using Redis. In choosing an optimal solution, it is critical to analyze usefulness of advanced Redis features for you. For video services, it is very effective to load preliminary statistics of video consumption (however other streams of events can also be loaded) in the Redis
processing lists to handle them to the systems of statistical analysis, billing, etc.

Databases Traditionally, many developers use MySQL and PostgreSQL, but highly loaded projects use to rapidly transfer many time-critical functions to non-relational databases, such as MongoDB and Cassandra. Non-relational databases have a potential of higher performance and lots of other functions needed by heavy projects: master-master replication, sharding etc. Video services often require statistics aggregation by a multitude of parameters over long periods. For such cases, it is important to have a separate database (usually, a relational one) to perform long and complex calculations with complex structural dependencies.