The quality of digital media depends on a multitude of factors, including, among other things, the bitrate, the frame rate, color range, pixel density, and processing. However, the actual number of pixels that fit in a frame, i.e., the resolution, is a key determinant of the clarity and definition of a digital image or video.
At Neverinstall, we stream applications over the internet to our users' browsers. Therefore, the stream's resolution plays a major role in determining our users' experience. While lower-resolution streams load faster and deliver better performance, they are blurred and often difficult to interact with. On the other hand, higher resolution may lead to lags, delays, and poor performance even though the stream is clear.
We had to address this challenge to make our platform a viable substitute for native machines.
Why is resolution important?
To understand the quality of digital media, measuring the definition becomes a base metric. For example, watching a video on YouTube at 1080p is much more desirable than watching it at 480p from a user experience perspective. Besides, high-resolution streaming also consumes more data bandwidth, so if you are on the last few gigabits of your mobile data package, you know it's time to turn down the resolution.
Resolution for an application stream
Although applications are streamed much like a video to a user's desktop, unlike a movie, users interact with different parts of the stream. Therefore, clarity becomes a major factor since users now need to clearly view several elements on the screen, including information, buttons, interaction points, and more.
An ideal user experience would require stable and consistent performance with decent clarity and definition for over 200,000 users, who are vastly a heterogenic group using different devices in different countries, and with varying internet speeds.
And here came the challenges.
The challenges
We realized that to deliver the best possible experience to our users, we had to mind the bandwidth consumption and constraints while delivering the crisp video quality. However, this was challenging since our platform runs applications on remote servers and streams them to the user's display. Here, internet speed and bandwidth define the quality of the stream of applications.
Therefore, we started looking at the problem from a different perspective and understood that we need the platform to adapt to different internet speeds and compensate for lower bandwidths without compromising on the steam quality.
This is where we started implementing a dynamic adjustment algorithm for optimizing resolution and bitrate based on the device's aspect ratio and the user's bandwidth. Here are some of the challenges that we faced along the way.
Challenge 1 - Aspect Ratios
In order to make the platform feel more native, we needed to give users a full-screen immersive experience. However, we had to achieve this across multiple devices to become truly accessible. Since different devices typically have different aspect ratios, we had to adapt to each situation and deliver the same quality regardless of the device of the user.
An allied challenge was the orientation of the device. We had to compensate for the portrait and landscape view alike for devices such as mobiles and tablets, and even for full-screen and a smaller window.
How we solved it.
We decided that we needed to fetch the information about the display on the fly. As a user creates and launches an application, we fetched details of the display of the user and used it to adjust the stream accordingly.
We used the aspect ratio as Parameter 1 for our dynamic adjustment algorithm.
Challenge 2 - Bandwidths
While developing the algorithm, we had to make a trade-off between bandwidth utilization and the crispness of the display stream. Although this may seem detrimental, improving the experience for users with slower internet speeds was necessary. Higher data consumption would evidently deliver higher quality, however, it would result in a lag for users with a slow internet connection. However, a significantly lower bandwidth utilization would result in a blurred-out stream.
How we solved it.
We decided that we need to limit the amount of bandwidth the application stream can utilize with our algorithm.
Consider this. If we utilize a user's complete available bandwidth, even a smaller session duration would affect the user's overall internet speed and data consumption. And on the other hand, if we restrict the utilization of the bandwidth too much, users with high-speed internet connections would not be able to get a fully immersive experience.
We were able to circumvent this problem by striking a balance. We built a speed test in-house to calculate the user's bandwidth with respect to the cluster they are operating in. And depending on the internet speed and available bandwidth, the algorithm automatically limits the bandwidth utilization to a certain percentage of the total availability.
We used the bandwidth Parameter 2 for our dynamic adjustment algorithm.
Challenge 3 - Bitrate
We had to make informed choices while deciding the bitrate for the stream based on the resolution dynamically opted for the user. This challenge was, however, more of an optimization of the existing solution and would be an outcome of the successful implementation of our algorithm.
Users interact with their applications at different points during a usage session in several ways. Now, every kind of interaction user interaction does not require the frames to be transmitted at the same and constant bitrate. For example, media consumption typically requires a high bitrate for a smooth experience. However, activities such as reading an article or coding do not.
How we solved it.
Due to users' changing bitrate requirements based on different interactions, we dynamically adjusted the bitrate based on the color complexity of a frame, internet speed, and aspect ratios.
A judgment call
We incorporated the parameters mentioned above into our algorithm. So, every time a user launches an app, our algorithm assesses these parameters and automatically decides which resolution and bitrate would best serve the user and deliver the best experience while simultaneously remaining mindful of their internet data consumption.
Manual control
Although dynamically adjusted resolutions are built to enhance user experience, we understand that some users may prefer more autonomy over the quality of their stream. Therefore, we added the feature to allow users to manually select the resolution of their application stream.
Suggested Read - Neverinstall Product Updates - May 2022
The way ahead
Neverinstall is constantly evolving and our team is trying to take user experience as close to native as possible. Subscribe to our blog to know more about the challenges we face and how we resolve them.
Sign up on Neverinstall now.
