WWDC21 takeaways for video developers

by Thijs Lowette

WWDC21 brings us SharePlay, AirPlay to macOS, and much more. Go through the bookmarks below to do your own WWDC21 OTT research, or keep reading to get the breakdown.

Bookmarks

Note: you may need to use the Apple Developer macOS application to view certain videos.

Let us know if we're missing important bookmarks!

1. Watch Together with Facetime

SharePlay is a Watch Together experience powered by FaceTime for all Apple devices.

  • Watch videos and live streams together with friends and family, while having a UI that allows you to pause/resume/seek the video for everyone.
  • This feature can also be used on a big screen through Apple TV.
  • "Group Activities" is the name of the framework powering SharePlay. ("A developer implements Group Activities to offer SharePlay to viewers.")
  • Viewers on the native apps must initiate the SharePlay session, but macOS Safari viewers can join them. Safari developers developers must leverage the Media Session API (+Coordinator API) to add support for SharePlay.
  • Apple is hoping that other macOS browsers will add support for their Coordinator/SharePlay API.
  • Refer to the bookmarks to go to the relevant documentation and WWDC21 video sessions.

It will be interesting to see how Google will respond to this. (Will we see a similar feature for Google Meet?)

2. Control playback Media Session API

Related to SharePlay, the Media Session API is now available on Safari for all Apple devices. This allows users to control their "media session" outside of their browser on iOS.

3. Insights of available variants

When creating an AVURLAsset asset with AVPlayer, developers can now query the "variants", granting them insights into the available bitrates, codecs, ...

Watch https://developer.apple.com/videos/play/wwdc2021/10143/ for more information.

4. Download configurations

Downloading video through AVPlayer has become more flexible. Developers can track the download task progress with NSProgress. Additionally, developers can set a "download configuration" to further specify what the download should do.

Watch https://developer.apple.com/videos/play/wwdc2021/10143/ for more information.

5. Easy Apple TV login

Viewers can now sign in to their Apple TV application with their phone through the Authentication Services API.

Watch https://developer.apple.com/videos/play/wwdc2021/10279/ for more information.

6. Spatial audio

You can get spatial audio on your Apple devices by providing multichannel audio in your HLS playlists (i.e.variant streams), or by sideloading them as tracks, or through the MSE WebKit API in browsers.

Watch https://developer.apple.com/videos/play/wwdc2021/10265/ for more information.

7. Transition media gaplessly with HLS

This feature can be used to seamlessly play advertisements, automatically play the next episode without loading, or offer "create-your-own-adventure" streams without buffering.

The connecting videos must share some traits, like DRM, codecs, sample rate, channel count, bit depth, as well as the same set of variants. CMAF should also be used.

Developers use the AVQueuePlayer API to implement this feature.

Watch https://developer.apple.com/videos/play/wwdc2021/10142/ for more information.

8. Picture-in-Picture changes

You can now automatically enter Picture-in-Picture when you push your app to the background through canStartPictureInPictureAutomaticallyFromInline.

Watch https://developer.apple.com/videos/play/wwdc2021/10290/ for more information.

9. Evaluate video with AVQT

You can inspect videos with the Advanced Video Quality Tool (AVQT). This is a "Perceptual Video Quality Tool" that compares your source video with your compressed video. This tool attempts to mimic how real people would rate video quality, and gives a score between 1.0 (=Bad) and 5.0 (=Excellent).

Watch https://developer.apple.com/videos/play/wwdc2021/10145/ for more information.

10. Interstitials and dynamic advertisements

Interstitials allow your manifest manipulator to refer to an additional HLS playlist to specify the advertisement. Your HLS media playlist (depicted below) can refer to this additional playlist through the EXT-X-DATERANGE tag. 

#EXTM3U
#EXT-X-TARGETDURATION:8
#EXT-X-PROGRAM-DATE-TIME:2020-01-02T12:00:00.000Z
#EXTINF:8
main1.0.ts
#EXTINF:8
main1.1.ts
#EXTINF:8
main1.2.ts
#EXT-X-ENDLIST

#EXT-X-DATERANGE:CLASS="com.apple.hls.interstitial",ID="ad1",START-DATE="2020-01-01T12:00:05.000Z",DURATION=15.0,X-ASSET-URI="http://example.com/ad1.m3u8",X-RESUME-OFFSET=0

There's a whole Interstitial API dedicated to it. It can also be used for client-side advertisements, and not just server-side advertisements.

How is this different from before? Before, the SSAI segments were inserted in the media playlist, and transitions were commonly announced with #EXT-X-DISCONTINUITY tags. This new approach is really tailored towards advertisements use-cases, while in the past there wasn't anything specific for it.

The specification is available at https://developer.apple.com/streaming/GettingStartedWithHLSInterstitials.pdf.

Watch https://developer.apple.com/videos/play/wwdc2021/10140/ for more information.

Other

Native macOS and Catalyst apps can now enter a "full" fullscreen, instead of just filling the active window.

AirPlay is coming to macOS.