Modelling Quality of Experience for Online Video Advertisement Insertion

The impact of online video advertisement has an evolving and undeniable influence on the success of online video streaming. A successful online video advertisement campaign deployment necessitates: “targeting appropriate marketing audience, determining optimum intervals to insert advertisement, associating the production quality of the content while considering advertisement conceptual features, matching the relevance of advertisement context to the content theme, calculating the applicable number of ads for stitching into the content, and correlating the ratio of advertisement length to total active watch duration”. This paper proposes a novel model for inserting advertisement into online video that considers content and commercial specific properties while optimizing Quality of Experience (QoE) by estimating suitable duration for advertisement, number of splits and content relation. The proposed model has been evaluated in a controlled on-line video test environment so that the success rate of this platform has been compared with the advertisement insertion strategies of technology frontrunners YouTube and Vimeo. In terms of medium and long length online videos, advertisements located within the content provides a better QoE compared to the ones that are located at the beginning of the video. For short length online videos, the general expectation of the audience tends to see the content immediately and any advertisement insertion related delay results in a corresponding customer behavior where 25% tend to quit after 3 seconds and another 25% after 5 seconds.


I. INTRODUCTION
S INCE the times of the first commercial television channels, advertising has always been a major component of the broadcasting life cycle [1]. From the beginning of last decade, we have been experiencing the transition from conventional single direction television transmission [2] to Internet-based return channel enabling content delivery technologies [34]. Additionally, the advertising methodologies have been also evolved rapidly in its own path including inventive applications such as automatic insertion [3] and virtual product placement [4] with an impact on advertisement campaign strategies and durations.
Comparing the necessary advertisement campaign duration to reach 40 million potential audiences, conventional television needs a 4-year of advertising [1] where standard Internet PageRank [5] based algorithms require approximately 6 months and social media with online video strategies [6], [7]. This may be accomplished in less than two months using machine learning to model customer tendencies and behavior. However, targeting wrong customers with a disturbing advertisement attitude have a negative impact on the users that might influence both the content delivery medium and the product/service that is being advertised [8]. A delay caused by either advertisement insertion [9] or a relatively long advertisement (compared to actual requested content) [10], might disturb user's overall Quality of Experience (QoE). Regarding this, the users might either end up quitting the watch session or lose their interest on the subject as discussed in [35]. Significant proportion of the databases for the major online video suppliers [11] such as YouTube or Vimeo consist of user generated content, which has either low resolution [12] or low production characteristics. A mismatch of content and advertisement resolution might also degrade user's QoE [11], [12]. Another aspect of advertisement insertion is the audio mixing levels [13] of the content and advertisement which might cause local audible peaks that will deteriorate QoE. Apart from audio and video mixing related issues, the number and context variety of advertisements [14] that are shown during a watch cycle has a major impact on the success of advertisement insertion. Showing the same advertisement repeatedly even for different content genres is generally defined as "intolerable" [15] by the online content consumer community. Moreover, the frequency of inserted advertisements and their duration play a major role in deteriorating [11], [12] overall QoE.
The aim of this paper is to design and develop a novel model for advertisement insertion in on-line video platforms. Unlike previous research works, particular emphasis has been given on the optimization of the advertisement insertion into video content by considering advertisement specific properties such as the ratio of the length of the advertisement to the content duration, the quality of the content and comparison to the advertisement content, the number of advertisements inserted into the content and location of the insertion while optimizing the QoE. This is accomplished by estimating suitable advertisement duration, number of splits and content relation.
The rest of the paper is structured as follows; Section II discusses the state of the art advertisement methodologies, Section III presents related works. Contributions are given in Section IV. Advertisement insertion parameters are clarified in Section V. In Section VI, details of the online video platform for advertisement insertion are discussed. Section VII provides QoE models for advertisement stitching and Section VIII debates the results. Finally, Section IX concludes the paper with future works.

II. STATE OF ART ADVERTISEMENT INSERTION METHODOLOGIES
On-line video streaming advertisement insertion related technologies can be classified into two main approaches; client side [12], [14] and server side [15], [17], [18]. Serverside advertisement insertion also known as "dynamic ad insertion" or "advertisement stitching" [15] takes place on the headend and the Content Management System (CMS) level [17] where the media is requested by the end-user and propagated through Content Delivery Network (CDN) as shown in Fig. 1. The content and advertisement videos are transcoded, audio mixed, mastered and normalized [13] and finally stitched together that forms a seamless single entity before being transmitted to the user. Frame accurate and video editing production capabilities are a must to ensure the flawless transition [19] between content to advertisement and visa-versa. This strategy requires many parameters [9]- [11] to be configured before the transmission of the content to the user. The number of advertisements to be stitched [10], where to insert the advertisement within the content [19] and which advertisement must be targeted to users [20], are the principal questions that must be answered. Although server-side advertisement insertion lacks the capability to act "on the fly" [16], single content-ad entity streaming has a capability to bypass advertisement filters [8], [15] whereas the origin of the content is initiated from a single source transcoder.
Client-side advertisement insertion [12] involves two or more independent sources for the content (originated from intermediate or edge cache CDN) [7] and the advertisement (generally from a third-party ad server) [14] as given in Fig. 2. These two different sources can be hindered by the widely available ad-block plugins [8] that are easily accessible through browser application stores [21]. These "generally" semi-intelligent ad block mechanisms [8] check if any browser module (in this case, the video player) tries to access content from multiple origins and provides a blocking mechanism in case of multiple origin access. This type of access blocking for the associated content might result as a disappointing watching experience where some part of the video content might either not be played properly or end up with consequences where some video chunks are not accessible by the video player [12], [15]. This type of effects will have a degrading impact on user's QoE [6] and eventually on the success rate of the content provider and CMS. An estimated number of users that have installed advertisement blockers has reached to 17% in 2017 reaching up to 32% by 2020 [8]. Predictions state that browser ad-blockers will result approximated 20 billion dollars degradation on advertisement revenue by 2020 [21].
Although client-side advertisement insertion seems to be vague when compared to server-side advertisement insertion [15], [18], in terms of implementation and operation, it is far simpler and requires less operational investment. Simplicity and easy integration capabilities makes client-side advertisement insertion the preferred solution for video campaigns. It is expected that by the end of 2019, client-side advertisement insertion solutions are expected to dominate the online advertisement with an estimated 74% sector size [16].
Overall, the primary drawback of the aforementioned implementations [6], [8], [12] is the lack of QoE influence and the disregarding the use of advertisement insertion parameters on the comprehensive user experience.
This paper proposes a QoE estimation methodology that is both applicable by server and client-side advertisement insertion systems. Parameters that have to be considered for advertisement insertion models include: the ratio of the length of the advertisement to the content duration, the quality of the content and comparison to the ad content, the number of advertisements inserted into the content, and location of ad insertion.

III. RELATED WORK
Yadati et al. has proposed an algorithm [22] to insert advertisements into video content using a brute force approach. The decision mechanism has been provided to hint the location of an advertisement to be inserted in Eq. (1) where AI(x i ) is the function to determine the advertisement insertion point, x i is the binary variables for advertisement insertion acceptance, A s (i), A s (i+1), max(A s ) are the validity score of the current scene, next scene and maximum arousal value, V S (i), V S (i+1), max(V S ) are relevance score of the current scene, next scene and maximum applicability value.
Eq. (1) [22] delivers an understanding of insertion locations throughout the content, principally based on the relevance of consecutive frames and the advertisement. In terms of a genrebased clustering attitude, the idea is promising. However, by following a theme-oriented classification approach, it is computationally complex to apply an "on the fly processing" to a large collection of advertisement content database, so that is practically quite difficult to implement.
Saito and Murayama has introduced a methodology [23], which contemplates user comments as a basis to indicate correct moment to stitch an advertisement to the content. Due to subjective nature of the approach, the user feedback provides useful understanding for the decision mechanism. However, the absence of a detailed QoE defiance results to the fact that this methodology does not conclude the subject that could be a guideline for the advertisement insertion.
Kodialam et al. has defined a formulation [10] for the decision of inserting an advertisement regarding the budget of the campaign defined in Eq. (2). The dual variables π(t) and δ(i) refer to the advertisement insertion at moment t with the bid b t (i, j) from the advertiser on user j and the remaining budget for each advertiser (i). Although this methodology introduces an understanding regarding budget for the campaign of advertiser, user clustering or relationship of content parameters are not very well clarified.
Ha et al. has provided [24] as a measure of willingness to continue watching an advertisement during online video consumption where τ s is the duration of commercial, ρ is the parameter that stands for willingness, N s represents the number of inserted ads in Eq. (3). Although the methodology provides a metric for user willingness and eventually QoE, it does not consider all aspects of advertisement insertion like ratio of advertisement to content or ad insertion frequency.
Le et al. has provided a formulization [9] to compare the gradual changes in bitrates and production quality to reduce negative impact on users, where B (i,j) refers to the bitrate of the content for i th chunk segment of user j in Eq. (4). Nonetheless, this method only compares the bitrate of consecutive content and not the whole watch session experience like option to skip the advertisement.
Wilbur has presented an estimation to measure user quit rate "PAZ" [25], which stands for either a zap or a power-off action that occurs during a commercial break after the viewer has been watching the channel for at least five minutes prior to the commercial break. Despite to the fact that the empirical estimation has been provided on a television broadcasting experience, channel and commercial breaks can be interpreted as online content and advertisement skipping capability.
According to [25], the authors focus on the estimation on a terminology based on the reverse version of zap, "PAZ" meaning: "the channel change incident that takes place after uninterrupted 5 minutes television-watching experience". Instead of the conventional zapping, noise associated with brief viewing events are filtered out which are unlikely to be related to advertising content in Eq. (5). B i represents the set of breaks, t ib is an indicator function which equals one when a PAZed break takes place and zero otherwise. B it is the subset of breaks in B i , which are not PAZed by i th user prior to available commercial slot t. The semi-parametric partial likelihood that user has a PAZed break b at slot t ib is given with L ib and eventually total quitting rate L i is achieved by evaluating L ib over the whole period of broadcasting in Eq. (6).
Unlike the works addressed in this section [9], [10], [22], [23], [24], [25], this paper provides an overall understanding of advertisement insertion metrics while considering its impact on QoE. All these parameters can be used as a guideline for any integrator to implement the decision mechanism for either server or client-side advertisement insertion module of an online video platform. To the best of our knowledge, this work is the first research paper with an overall understanding of advertisement insertion and its impact on QoE for any online video service. Additionally, this paper makes an analysis and study on the content characteristics and its relation to advertisement resolutions and encoding type.

IV. CONTRIBUTIONS OF THIS PAPER
The aim of this paper is to formulate a QoE model for advertisement insertion for an online video content with different fundamental properties to provide an advertisement stitching guideline for any online video service.
To be able to proceed with validation for the QoE models, an online video platform with ad insertion capability has been developed. The properties of this system are listed as follows: 1) The platform executes via a hybrid architecture of Docker & Virtual Machine (VM) on Amazon Web Services (AWS) and available for public access through "www.utkubulkan.co.uk/ad.html". The application layout for the advertisement service VM instance has been provided in Fig. 3.
2) The capabilities of the portal include random advertisement insertion to a catalogue of video content. During or before the content, depending on the ad-content relationship, advertisement skipping can be offered to the user. At the end of each watch session, which might include either single or multiple advertisements along the content, the user is queried with a questionnaire regarding their experience with the online video platform. A screenshot from the online video player with additional debug information regarding the statistics has been presented in Fig. 4. 3) Advertisement and content related parameters are stored in a database along with the user experience survey. The subjective user data is correlated to objective parameters and the models and the parameters inside the models are based upon these data. 4) The comparison and advantages upon already established work has been presented in Section III while error analysis in Section VII, where the outstanding aspects of the models in the work have been underlined.

V. ADVERTISEMENT INSERTION PARAMETERS
In this section, objective parameters that are closely associated to advertisement insertion metrics are going to be introduced to assemble a foundation for formulizing ad insertion QoE models in Section V.

A. Location of Advertisement Stitching Into Content
Conventional television broadcasting inserts commercials during the show time [2] of programs and/or in between different programs. However, online video has modified this practice [7] by showing advertisements just before the content due to the nature of short content duration of online video [16]. This is due to the fact that users intend to quickly consume 3-5 minutes content and tend to quit just afterwards.
During the evolution of on-line media platforms such as YouTube or Vimeo, where the long period videos became more frequent [18] advertisement stitching during the content suited in a better way where users spend more time which is close to a traditional television experience. Due to these facts, analyzing the length of the content and creating a decision mechanism for advertisement stitching location plays a major effect on user's QoE.

B. Ratio of Length of the Content and the Advertisement
To achieve a successful advertisement insertion, another important measure is the ratio of the content length to the advertisement length [21]. A short length content (30 seconds to 2 minutes) proceeded by a 2 minutes advertisement would disturb the session quality [19]. In the same context, showing 10 seconds advertisements in every minute for a medium length content (2 to 10 minutes) can also degrade QoE and user will more likely tend to quit.

C. Advertisement Insertion Frequency
For longer duration content, which is quite common on YouTube and even Facebook nowadays, advertisement insertion frequency is one of the major considerations that decide the success of advertisement insertion. Currently, online video broadcasting sector standards tend to show an advertisement in every ten minutes for long content (10 minutes to 2 hours), where the advertisements are marked on the player timeline [16].

D. Comparison of Production Quality of Content and Advertisement
Due to the nature of advertisement, any commercial attempt to promote a product requires an investment, especially on the production of the advertisement video. This will probably lead the production company to provide commercial television quality advertisements.
Nevertheless, a massive amount of the content that is served by online video platforms are made up of user generated low resolution and/or low production quality content. A mismatch of quality might influence the overall watch experience.

E. Skippable Ads, the Decision for Length of the Non-Skippable Duration
For any online content service or social media provider, to satisfy user demands, many user models and algorithms [12], [16], [18] run in background to decide which advertisements must be shown for content genre. According to the advertisement campaign and marketing payment choice, the advertisements are generally skippable including a nonskippable duration. The length of the advertisement plays a major role in this decision process, especially on the non-skippable interval.

VI. ONLINE VIDEO PLATFORM FOR ADVERTISEMENT INSERTION
In order to gather information from users, an online video platform has been developed [33]. The platform is capable of streaming a wide range of online video content with a collection of ads alongside that can be dynamically stitched into watch session. At the end of each session, users are questioned by a subjective user survey which includes information regarding their experience from the service. Instant user feedback is an advice mechanism that is being employed for nearly every major Web service including YouTube, WhatsApp and Facebook [16]. In this research work, the users are queried about their experience regarding the parameters that are given in Section V; the relevance of the advertisement, the impression of advertisement length vs content length, the influence of the skip ad option duration and finally, the impact of the location of the advertisement inserted to overall experience. An example survey has been presented in Fig. 5. The strings "very bad, bad, moderate, good and very good" means a numerical value that ranges from 1 to 5 and throughout the paper this concept might be referred as numbers or with the mentioned strings.

A. Crowdsourcing, the Method of Collecting Subjective User Experience
Crowdsourcing [26] has been selected to collect data for this experimentation. Due to its flexibility, wide geographical distributed and informal data collection ability, crowdsourcing has shown good performance against lab-based experiments [27] especially for online Web services. The nature of crowdsourcing gives the programmers, testers and convincingly the users a real-life utilization experience while still holding a trial utilization feeling. This gives the unperturbed chance to proceed with Continuous Integration (CI) and Continuous Deployment (CD) attitude while making the necessary software ready and tested before deploying to the field.
From a crowdsourcing point of view, in this work, subjects have been requested to contribute through a remote assessment technique via the online video platform link that has been presented in Section IV, where they have provided their experience with the advertisement insertion system through the subjective metrics in Fig. 5. This information has been captured and error estimation have been evaluated resulting into a comparison table for a variety of content parameters.
The methodology that is employed in this work, is based on a platform allowing the users to watch online video content through the Web service accessing random video contents with their smart devices (listed in Section VI-B) where advertisements have been inserted into their watch experience. At the end of each video session, users have been provided a survey that consists of questions regarding to the relevance of the parameters of the content and inserted advertisement.

B. Subjects, Equipment and Test Content
Subjects who have participated in the research are undergraduate and postgraduate students attending computer science and data science programmes at London South Bank University at the time of the experimentation. A total of 24 test subjects have participated for the testing evaluation in 3 different 60 minutes sessions. Testers have used 12 different consumer devices including a variety of mobile phones; Samsung S3, S4, S5, Note 3, Note 4, Sony Xperia XZ which have resolution of 1920x1080, HTC 10 (2560x1440) and personal computers; Dell Latitude e6410 (1280×800), Macbook (2560x1600), HP Elitebook8460 (1366x768), Probook 430 (1366x768) where either Firefox or Safari browsers have been executed depending on the operating system of the particular device. All test consumer equipment that has been used via crowdsourcing received service from the proposed video and Web services that executes on Amazon Web Services (AWS) EC2 cloud platform running via Docker and In terms of test content, the subjects have been provided a collection of 10 three minutes, 10 three to twenty minutes and 10 twenty plus minutes as video content catalogue. Additionally, a separate catalogue of 30 different publicly available advertisement content ranging from 30 seconds to 2 minutes have been used. Information regarding some of the selected videos and advertisement content has been presented in Table I. All the content can also be accessed through the online video streaming portal that is associated with this paper. Relevant access information has been provided in Section IV.

VII. QOE MODELS FOR ADVERTISEMENT STITCHING
In this section, the methodology to model "QoE for advertisement insertion" is presented. This model considers the advertisement campaign parameters such as content duration, advertisement duration, user's total watch session and number of stitched advertisements. Finally, Algorithm 1 provides a decision mechanism that calculates QoE according to the position and frequency of the advertisements that are going to be stitched into the content regarding the T MAX total watch session duration for a user. The list of notations regarding the formulas for the rest of the manuscript has been declared in Table II.
Classification of content duration plays a major role for the decision of advertisement insertion mechanisms. Any type of content (either broadcast production quality or user generated) can be classified as short content if the actual duration of the video is less than 3 minutes. Music clips, funny videos, short information-oriented content fall in this range. According to advertisement stitching conventions [16], generally only one advertisement is inserted to short duration content.

A. "λ", the Ratio of the Ad Duration vs Content Duration
Let's denote, "λ", the ratio of the advertisement duration "l a " is the content duration and "l c " is the advertisement duration which must be properly small where "l c " must also be greater than l a . This constraint is given with Eq. (7). As shown in Fig. 6, the relationship of QoE for Advertisement Insertion vs "λ" has been given for different T max values. This is going to be discussed in detail within the following paragraphs.
Considering the case for very short durations [15,25], any content spanning in less than 20 seconds, usually is out of advertisement scope unless the user has continually requested content of this duration and genre. λ = l a l c , l c > l a .

B. Nature of QoE for Advertisement Insertion Function
The exponential nature of the QoE AD function originates from the foundational underlying behavior of the user's watch session quitting probability distribution also named by YouTube as "audience retention graph" [31], which has an exponential with a negative power behavior in nature.
A user that has a guaranteed decision to watch a content has the normalized maximum attention probability for the beginning of the content and minimum attention probability at the end of the content [32]. This creates an asymptotic behavior that is tangent to the relevant axis regarding the edge and time conditions of the watch session.
This whole picture of exponential nature of user attention analysis hints for the successful advertisement strategy as imitating capability of the "exponential function". This strategy provides a basis to insert advertisements within the time window of the user's attention. The coefficients of the exponential function shape the behavior of the curve regarding the properties of the video and the advertisement.

C. Constraints for T max <3min
In Section VII-B, the exponential nature of QoE for advertisement insertion has been explicitly discussed. Regarding the fact that, the disturbance in user's attention has been described in an exponential behavior, the QoE function for advertisement insertion have also been formulated as natural base exponential functions in Eq. (8), Eq. (12) and Eq. (13).
For λ ≈ 1, n = 1, T max < 3min, in the case of the consecutive watch behavior for the very short content with parameters, QoE AD can be determined by Eq. (8) where κ refers to constant which normalizes the QoE, l c to content duration, l a to advertisement duration, φ is the duration for skippable advertisement capability and T max is the longest runtime for the content in this duration classification.
Generally, content with a runtime duration less than 3 minutes (T max < 3min), online video streaming conventions only encourage insertion of a single advertisement to keep audience interest undisturbed. In Eq. (8), this is denoted implicitly with "n=1" where n is the constant value representing the number of advertisement insertion.

D. Constraints for T max ≈ 10 min
Any online content with duration between 3 to 10 minutes is a good candidate [7] to have multiple advertisements stitched during a video watch session. News, short movies and Web blogs are examples of this categorization.
For A λ < 1, T max ≈ 10min, "n", is the number of advertisements and is defined by the ratio of content length to maximum content duration as an integer via ceiling function in Eq. (9). n = ceil l c T max (9) The whole watch experience session duration "L" is defined by adding each advertisement duration l ai along with the content duration l c , as given to Eq. (10).
In order to formulize QoE AD for the content in this range, average A λ has been introduced in Eq. (11). As a supposition, Eq. (12) has been presented to model QoE in case of T max = 10min.
A λ = n i=0 l ai l c (11) The symbol "β" represents the weight for the relevance of the advertisement to the content. Similar to methodologies that frontrunner online video systems follow, each video and advertisement content are tagged and associated with 4 words. Regarding the number of matching tag words, any advertisement that is relevant to target content has a higher corresponding "β" value where it is a normalized entity between 0 and 1.
Movies, documentaries, full concert videos are regarded as very long content. Although they are quite frequent and mainly form the foundation of conventional television broadcasting [1], [2], methodologies for handling very long content and advertisement stitching are fairly new in online video domain [11]. Nevertheless, they constitute an unquestionably important portion of today's OTT video demand by more than 40% [16]. Following the advance of streaming services such as Netflix, Amazon Video and Hulu [15], online video has switched from short video experience to a television like experience. Due to the current operational similarities to television broadcasting, very long online content can also host multiple ad insertion points without causing deterioration in QoE [18].

E. Constraints for T max ≥ 60 min
For A λ 1, T max ≥ 60min, Eq. (13) approximates the QoE as a function of "t ai " which represents the advertisement stitching moment during a watch session. The symbol "μ" represents the comparison of the production quality of the advertisement to the quality of the content. The closer the value to 1, the closer the production quality is.
The production quality for the range of very long content generally is very high as this classification consists of cinema movies and television targeted generated content.
t ai Tmax +lc L (13) Due to this foundational distinctive implication, the advertisement quality plays a characteristic role for user experience. When this impact is considered, a better understanding of QoE for very long content can be achieved.
From the overall point of view, QoE for advertisement insertion has been associated with advertisement insertion frequency and duration of the watch session (in minutes) in Fig. 7. As the advertisement insertion frequency increases, the user's QoE for advertisement insertion drops sharply for content shorter than 10 minutes. However, for longer duration content (ranging from 20 to 40 minutes), user's tolerance for acceptable advertisement increases and this creates a much Algorithm 1 QoE Estimation for Ad Insertion PREREQUISITES: l c , l a , τ , λ, L, n, T max , 1. WHILE (T < CAMPAIGN DURATION) 2. FOR EACH AD, COMPUTE RATIO OF λ = l a /l c , A λ and L 3. IF λ ∼ 1, COMPUTE EQ (8). 4. ELSE IF A λ < 1, COMPUTE EQ (11), (12). 5. ELSE IF A λ << 1, COMPUTE EQ (13). 6. IF QOE AD = QOE AD (T 1 )-QOE AD (T 2 ) < E QoE THEN RECALCULATE λ, L, N. 7. END WHILE. more suitable setting to introduce short but frequent and succesful advertorial breaks.
Generally, the longer content means longer user engagement. This is reflected from the value of the content producer. Still, when the content duration exceeds 50 minutes, audience tend to lose attention due to the longevity of active watch session and consequently each additional advertisement creates a high draining impact on QoE.

F. Algorithm to Calculate QoE for Advertisement Insertion
The following methodology given in the Algorithm 1, states that any advertisement insertion algorithm must follow this pattern to cover up the necessities of content and advertisement property comparison. The content duration is the primary indicator to switch between different QoE models for advertisement insertion. Due to the nature of online video platforms, there is a requirement to support a library of content with a variety of different duration.
One of the primary key idea that has been introduced with this paper to overcome the failure of understanding of differences in content properties [9], [22], [25] and corresponding user expectations [10], [11].
Depending on the content to advertisement interval ratio, Algorithm 1 chooses a "duration comparison oriented" advertisement insertion strategy that should be employed for a successful advertisement campaign deployment.

VIII. COMPARISON AGAINST TECHNOLOGY
FRONTRUNNERS, YOUTUBE AND VIMEO Advertisement strategies of major technology frontrunners change rapidly to fulfill the demand and match the corresponding watching habits of the consumers. In terms of content quality and genre, Vimeo targets original creators where in YouTube both user generated content along with premium content. As each of these companies use their own proprietary algorithms to insert advertisement into content, it is hard to represent their advertisement stitching strategy within a controlled test environment. Yet, it is not impossible to compare the results of the proposed model against these major corporations; insights and observable details of the advertisement insertion strategies of these companies will be shared in this section.

A. YouTube
YouTube follows a variety of different ways to achieve their advertisement deployment. In terms of companies, which want to advertise their content, a portal is provided where target sectors and contents can be specifically declared to match the customer profile that is indented to reach. In terms of content creators, a wide range of options are available where the number of advertisements, the content quality and relevance, even the bid for each possible successful engagement can be tuned. The overall output of all these possible combinations brings out an interface where the users can create playlists and a continuous watch session experience.

B. Vimeo
Vimeo provides a platform where only high-quality user generated content is broadcasted. This results in a different fashion in terms of advertisement stitching than other platforms due to the strict relevance of advertisement context to the content. Still, to have a validated test environment, a playlist with similar content has been created and uploaded to test against Vimeo's advertisement insertion strategy.

C. Comparison of the Proposed Model Against YouTube & Vimeo
To be able to compare the success rate of QoE for advertisement insertion of this paper's approach versus YouTube & Vimeo, a controlled test environment has been used.
Creating a similar YouTube experience for the audience is quite important and relatively hard to achieve. To accomplish this, associating and comparing the success rate of an equivalent playlist methodology has been followed. This paper uses a playlist for video content and a separate relevant playlist for advertisements. Following an analogous pattern, a playlist with same video content properties has been created with YouTube and Vimeo user accounts.
Obviously, both YouTube and Vimeo handle advertisement insertion mechanisms themselves and therefore the properties of the playlist is maintained. The internals of advertisement insertion are done natively using their own self-regulating algorithms. Yet, from this controlled test environment point of view, these three equivalent playlists (proposed, YouTube's and Vimeo's) are provided to the test subjects as shown in Fig. 8. Following this, the subjects have been kindly asked to provide relevant evaluation about the experience with their For the sessions that has been provided by YouTube and Vimeo, their proprietary algorithms insert advertisements and related QoE subjective data are collected from the users manually, yet, keeping in accordance with the QoE questionnaire that has been given in Fig. 5. This comparison technique provides an appropriate setting for overall understanding of advertisement insertion effects on online video streaming and QoE.

IX. RESULTS AND DISCUSSIONS FOR THE QOE MODELS
In this section, QoE models for projecting user experience depending on the content parameters will be compared to the already established work related considering advertisement insertion methodologies for video delivery systems while considering performance metrics of other online streaming platforms. The QoE error analysis for different duration parameters have been presented in Table III and following that the detailed error analysis has been discussed throughout Section IX-B.

A. Error Comparison for the Models
Regarding calculated and actual qualitative values, the error has been measured [29] with three different methods: Pearson correlation, root mean square error and mean average error.
The Pearson correlation measures the linear association between a model's performance and the subjective QoE. Root mean square error is the square root of the average of squared errors. Mean average error provides a simple analysis of the average difference between prediction and real values. The error is proportional to the absolute difference of actual and calculated.

B. Performance Comparison
The proposed QoE AD model Eq. (8) that is valid for short duration content which has T max ≈ 3min and λ ≈ 1 shows better performance than Ha et al. [24] and Le et al. [9]. Due to the nature of the user expectations from a short duration content, the impact of skippable advertisement duration capability "φ" plays a major role as the users are generally keen on to quickly access content, consume it and quit afterwards. Online video services employ different durations applicable to different length content durations varying between 5, 10 and 15 seconds [27]. Other models [9], [24] lack the impact of this capability, hence the performance of the Eq. (8) presents a better understanding of similarity to user experience in terms of all the error metrics PCC, RMSE and MAE.
QoE AD model Eq. (12) for T max ≈ 10min provides a good understanding of medium length content, which covers news and Web blogs as example. The impact of content to advertisement relevance represented with symbol "β" distinguishes the model that is provided in this paper and furthermore this provides a better understanding of user experience when compared with other established works. Comparison table gives lower error and better results for relevance parameter β > 0.5 which samples the occasions with higher relevant content ad selection.
QoE AD model for T max ≥ 60min offers a representation for very long content and the impact of advertisement production quality with the symbol "μ". When low quality advertisements are inserted during high quality long duration content, the users are generally annoyed, and this results in a corresponding QoE deterioration. For μ < 0.5 where the advertisement quality cannot match high content quality, the approximation for QoE shows a good error level when compared to μ > 0.5. YouTube's TrueView in-slate advertisements [30] are of this kind where long-form YouTube content are shown to users with high quality ads matching the quality of YouTube Partner videos.
From an overall error analysis point of view, the models introduced in this work provides a better understanding of QoE when compared to the works that are available in academic literacy as presented in Fig. 9. Although Ha et al. [24] and Le et al. [9] have provided a general understanding of advertisement insertion and its influences on some extent, the parameters that are introduced in this paper cover different range of durations, shows better performance in all cases in terms of PCC, RMSE and MAE metrics.

C. Complexity Comparison
This paper presents a novel advertisement insertion methodology to deliver increased QoE. The proposed methodology is based on Algorithm 1 where QoE AD for advertisement is estimated in Steps 3, 4 or 5 by determining the relative advertisement parameters. Following that, at two consecutive timestamps, overall QoE is measured via the incremental chance in movement. Regarding this outcome, the arguments in step 2 are recalculated if QoE difference is greater than ε QoE . The loop that executes during the campaign duration has a complexity of O(n) with the best case scenario where the condition in Step 6 is met in the first place where the loop is traversed through only once during the campaign duration. Worst case complexity would result in O(nlog(n)) where for each iteration the parameters regarding the advertisement insertion should be recalculated to prevent QoE deterioration. Even though the whole watch session experience is not considered by both the methodology that Le et al. [9] and the work of Ha et al. [24], still, the complexity is O(n 2 ) for estimating QoE for the impact of advertisement insertion. Additionally, these works provide a momenteraily understanding of QoE and preliminary is not based on recalculation regarding the feedback from QoE. Regarding these facts, the methodology that this paper provides has a better standing point for enhancing QoE for advertisement insertion when compared with the rest of the other methods that has been mentioned throughout the manuscript.
As a final note, due to the proprietry nature of advertisement insertion mechanisms of both YouTube and Vimeo, commenting on the complexity of their algorithms is quite difficult.

X. CONCLUSION AND FUTURE WORK
In this work, a general overview of advertisement insertion technologies and the impact of advertisement insertion parameters on QoE have been discussed. According to results, the advertisements that are shown during the online content showed better results, however content relevant advertisement insertion provided an evident interest and high QoE on the users. Short duration or skippable advertisement insertion preceding the content showed a small disturbance on QoE, however the delays occurring at client-side ad insertion or static ads provide a poor QoE and declared as showstopper by many of the subjects.
Conclusively, the equations and algorithm that are provided throughout the paper can provide a basis for a guidance to implement an advertisement stitching component for an online video service where the choice can either be client or serverside advertisement insertion.
As a future work, a user centric history-based recommendation system is planned to be implemented. In addition to video and advertisement specific parameters, a contextual object detection system is envisioned to be developed. In this framework, state of art, publicly available, general purpose RCNN TensorFlow trained models will be used for object detection. These machine learning models are validated by their wide use and outstanding performance even on limited resource cloud systems. Object detection will provide labelling for the video and advertisement content and describe them with related tag metadata. Association of the metadata will provide a better understanding of interest area of a user or a cluster of users which will enhance the overall QoE regarding the advertisement insertion. Improved QoE will increase the user watch session time and overall performance of the video delivery system by providing better profits and higher advertisement match rates.