KlugMedia documentation

Introduction

KlugMedia moves real-time media out of the AIR runtime and into one native session per app. Camera capture, H.264 encoding, RTMP publishing, H.264 decoding and display, microphone capture, Opus encoding and decoding, and echo cancellation all run natively. ActionScript sends commands and receives status events; no audio or video frames pass through the AIR stage.

• Video out: the camera is captured and published as H.264 over RTMP, with optional publish login, live bitrate changes and automatic bitrate adaptation on weak networks.
• Video in: a remote stream is received, decoded and drawn natively in a rectangle above the AIR stage: aspect-fit, stretched or crop-to-fill with rotation.
• Camera controls: native preview, camera switch (front and back, built-in and USB cameras on the Mac), torch, real camera zoom and JPEG snapshots.
• Voice: two-way Opus voice with the platform's echo cancellation, both sides talking at once, carried as data messages over your RTMP connection.

Video is sent as classic H.264 FLV tags (no enhanced RTMP), so it works with standard RTMP servers, including older ones. The voice relay needs a server that forwards NetStream.send() data messages to subscribers; see Server requirements.

Requirements

Add the extension

1 · Copy the ANE

Copy com.klug.media.ane into the extension folder of your project (for example lib/ or extensions/) and add it to your build: in your IDE as a native extension of the project, or with -external-library-path for compiling and -extdir for packaging on the command line.

2 · Declare the extension ID

<extensions>
    <extensionID>com.klug.media</extensionID>
</extensions>

The camera part, the voice part and the licence call all live in this one extension. Do not add separate camera or echo-cancellation extension IDs.

3 · iOS and iPad apps on the Mac

Add the two usage descriptions. iOS shows them in the permission prompts; without them the system ends the app on the first camera or microphone access. No entitlements and no background modes are needed.

<iPhone>
    <InfoAdditions><![CDATA[
        <key>MinimumOSVersion</key>
        <string>17.0</string>
        <key>NSCameraUsageDescription</key>
        <string>Your text: why the app uses the camera</string>
        <key>NSMicrophoneUsageDescription</key>
        <string>Your text: why the app uses the microphone</string>
    ]]></InfoAdditions>
</iPhone>

The same iOS build runs as an iPad app on Apple silicon Macs. Nothing else is needed in the descriptor; make the iPad app available on the Mac in App Store Connect.

4 · Android

The extension needs camera, microphone, audio-settings and network access. WAKE_LOCK is recommended: with it, the extension keeps the Wi-Fi radio and the CPU awake while it publishes, which avoids stalls on Wi-Fi power saving; without it streaming still works. Hardware acceleration must be on (it is the default) because the crop-to-fill receive mode draws into a TextureView.

<android>
    <manifestAdditions><![CDATA[
<manifest>
    <uses-sdk android:minSdkVersion="23"/>

    <uses-permission android:name="android.permission.INTERNET"/>
    <uses-permission android:name="android.permission.CAMERA"/>
    <uses-permission android:name="android.permission.RECORD_AUDIO"/>
    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>
    <!-- recommended: Wi-Fi and CPU stay awake while publishing -->
    <uses-permission android:name="android.permission.WAKE_LOCK"/>

    <uses-feature android:name="android.hardware.camera" android:required="true"/>
    <uses-feature android:name="android.hardware.microphone" android:required="true"/>

    <application android:hardwareAccelerated="true"/>
</manifest>
    ]]></manifestAdditions>
</android>

No activities, services or receivers are needed. CAMERA and RECORD_AUDIO are runtime permissions: request them in the app before the first start (see Permissions).

5 · macOS

Add the usage descriptions to the descriptor. macOS ends the app on the first camera or microphone access when they are missing.

<macOS>
    <InfoAdditions><![CDATA[
        <key>NSCameraUsageDescription</key>
        <string>Your text: why the app uses the camera</string>
        <key>NSMicrophoneUsageDescription</key>
        <string>Your text: why the app uses the microphone</string>
    ]]></InfoAdditions>
</macOS>

When you sign the app for the Mac App Store (sandboxed) or with the hardened runtime, the entitlements file you sign with needs network client access, the camera and audio input:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <!-- Mac App Store: sandbox -->
    <key>com.apple.security.app-sandbox</key>
    <true/>
    <key>com.apple.security.network.client</key>
    <true/>
    <key>com.apple.security.device.camera</key>
    <true/>
    <!-- microphone: sandbox key -->
    <key>com.apple.security.device.microphone</key>
    <true/>
    <!-- microphone: hardened runtime key -->
    <key>com.apple.security.device.audio-input</key>
    <true/>
    <!-- plus the entitlements the AIR runtime itself needs for your signing setup -->
</dict>
</plist>

The extension only opens outgoing connections; no server entitlement is needed.

Getting started

The first working publish takes four steps: set the licence key, check isSupported, listen for status events, start. Every call returns at once; the outcome arrives as a flash.events.StatusEvent on the same object.

import com.klug.media.KlugMediaLicense;
import com.klug.petcam.camera.CameraStreamANE;
import flash.events.StatusEvent;

// 1 · once at startup, before any streaming (key delivered with your purchase)
var licensed:Boolean = KlugMediaLicense.setLicenseKey( "YOUR-LICENCE-KEY" );

// 2 · the camera and video part is a singleton
var media:CameraStreamANE = CameraStreamANE.instance;

if ( media.isSupported )
{
    // 3 · results arrive as StatusEvents
    media.addEventListener( StatusEvent.STATUS, onMediaStatus );

    // 4 · publish (camera permission already granted, see Guides)
    if ( !media.startPublish( "rtmp://your.server:1935/live", "camera-1" ) )
        trace( "start refused: " + media.lastError );
}

function onMediaStatus( e:StatusEvent ):void
{
    if ( e.code != CameraStreamANE.STATUS_EVENT ) return;   // "cameraStatus"

    if ( e.level == "connected" )
        trace( "live" );
    else if ( e.level.indexOf( "failed:" ) == 0 )
        trace( "failed: " + e.level.substr( 7 ) );
}

With the defaults, startPublish() sends the back camera at 640×360, 12 fps, 800 kbps, with a keyframe every 2 seconds. The other side plays the stream with startReceive() (Receive a stream), and voice is added with OpusVoiceBridge (Two-way voice).

Guides

Licence key

Call KlugMediaLicense.setLicenseKey() once, early at app startup and before any publish, receive or voice start. The key is delivered with your purchase and is bound to the application ID you entered at checkout, so it only works in that app. The call returns true when the key is valid for the running app.

import com.klug.media.KlugMediaLicense;

if ( !KlugMediaLicense.setLicenseKey( LICENCE_KEY ) )    // LICENCE_KEY: your own constant
    trace( "KlugMedia licence key not accepted for this application ID" );

Permissions and the one capture owner

KlugMedia owns the camera and the microphone of the process. Two captures of the same device fight over it, and on iOS a second microphone reconfigures the audio session and degrades echo cancellation. So:

• Do not call Camera.getCamera(), Microphone.getMicrophone() or Microphone.getEnhancedMicrophone() while the extension runs. Even reading such a getter creates an AIR capture object.
• Use cameraCount instead of AIR's Camera.names, and getCaptureLevel() instead of an AIR microphone's activityLevel.
• Request the permissions before the first start, through AIR's permission managers: they do not create capture objects.

Platform behaviour: on iOS the voice start() does not raise the microphone prompt itself; it fails with the error "microphone permission undetermined" until permission was granted. On Android camera start fails with failed:<reason> and voice start with "RECORD_AUDIO permission is required" without permission. On macOS the extension raises the camera and microphone prompts itself when the camera first starts; a voice start() without a decision raises the prompt, fails once, and sends micPermission when the user agreed: then call start() again.

import flash.events.PermissionEvent;
import flash.media.Camera;
import flash.media.Microphone;
import flash.permissions.PermissionManager;
import flash.permissions.PermissionStatus;

// the permission managers do not create AIR Camera or Microphone objects
var camPm:PermissionManager;
var micPm:PermissionManager;

function askPermissions():void
{
    try { camPm = Camera.permissionManager; micPm = Microphone.permissionManager; }
    catch ( e:Error ) { startMedia(); return; }   // platform without permission managers
    askCamera();
}

function askCamera():void
{
    if ( camPm.permissionStatus == PermissionStatus.GRANTED ) { askMicrophone(); return; }
    camPm.addEventListener( PermissionEvent.PERMISSION_STATUS, function( e:PermissionEvent ):void {
        if ( e.status == PermissionStatus.GRANTED ) askMicrophone();
        else showPermissionHelp();                 // your own screen
    });
    camPm.requestPermission();
}

function askMicrophone():void
{
    if ( micPm.permissionStatus == PermissionStatus.GRANTED ) { startMedia(); return; }
    micPm.addEventListener( PermissionEvent.PERMISSION_STATUS, function( e:PermissionEvent ):void {
        if ( e.status == PermissionStatus.GRANTED ) startMedia();   // your own function
        else showPermissionHelp();
    });
    micPm.requestPermission();
}

Publish a camera stream

startPublish() opens the camera, starts the H.264 encoder and publishes to url/streamName. Pass the server URL without the stream name; the extension joins the two. It returns true when the start was accepted; the connection result follows as cameraStatus events: connecting, then connected or failed:<reason>.

import com.klug.petcam.camera.CameraStreamANE;
import flash.events.StatusEvent;
import flash.events.TimerEvent;
import flash.utils.Timer;

var media:CameraStreamANE = CameraStreamANE.instance;
var retry:Timer = new Timer( 5000, 1 );

media.addEventListener( StatusEvent.STATUS, onStatus );
retry.addEventListener( TimerEvent.TIMER_COMPLETE, function( e:TimerEvent ):void { goLive(); } );

function goLive():void
{
    media.startPublish( "rtmp://your.server:1935/live", "camera-1",
                        640, 480,                       // 4:3, the full camera picture
                        12, 800, 2,                     // fps, kbps, keyframe interval
                        CameraStreamANE.CAMERA_BACK,
                        null, null );                   // no publish login
}

function onStatus( e:StatusEvent ):void
{
    if ( e.code != CameraStreamANE.STATUS_EVENT ) return;
    var level:String = e.level;

    if ( level == "connecting" )                     showBusy();
    else if ( level == "connected" )                 showLive();
    else if ( level == "disconnected" )              showOffline();
    else if ( level.indexOf( "authError:" ) == 0 )    askForLogin();
    else if ( level.indexOf( "failed:" ) == 0 )
    {
        trace( "publish failed: " + level.substr( 7 ) );
        media.stopPublish();          // clean stop before the same name is published again
        retry.reset(); retry.start();
    }
}

While publishing, the extension keeps the device awake: on iOS the screen does not lock automatically, on Android the Wi-Fi and CPU locks are held (with the WAKE_LOCK permission). Both are released when publishing stops.

Stream orientation

streamRotation decides how the published picture is oriented. The default 0 publishes a landscape picture that follows the landscape side the device stands on, so the picture stays upright when the device is turned over by 180 degrees. On Android the extension restarts the publish for this turn by itself (you see connecting and connected again). 360 publishes an upright portrait picture; 90, 180 and 270 turn the landscape picture clockwise. The front camera is not mirrored in the stream.

// a device standing in landscape: default, follows the side it stands on
media.startPublish( url, "camera-1", 640, 480, 12, 800, 2, CameraStreamANE.CAMERA_BACK, null, null, 0 );

// a phone held upright in a portrait video chat: upright portrait picture
media.startPublish( url, "chat-1", 640, 480, 8, 600, 2, CameraStreamANE.CAMERA_FRONT, null, null, 360 );

Fixed rotations depend on how the device is held; test them on your target devices.

Stop and restart cleanly

Always call stopPublish() before you publish the same stream name again, for example after a network change or a failed: event. Some servers keep the old stream for a while when a connection simply drops, and then refuse or misroute the new one. A clean stop releases it at once.

• A second startPublish() with exactly the same URL, name, size, frame rate and camera while that stream is running is ignored and returns true. Any other change stops the running stream and starts the new one.
• stopPublish() stops the encoder and the connection and sends disconnected. The camera keeps running while the preview is shown; otherwise it is released.
• stopReceive() closes the playing stream on the server and removes the native view.
• Stopping is safe at any moment, also while a stream is still connecting.

function restartPublish():void
{
    media.stopPublish();                     // always first
    media.startPublish( url, "camera-1" );    // same name again: now safe
}

Receive a stream

startReceive() plays a remote stream, decodes the H.264 video natively and draws it in a native view at the given rectangle. Coordinates are stage coordinates (the same values as stage.stageWidth and stage.stageHeight); the extension converts them to the platform's units. The view is transparent until the first frame arrives, so your AIR background shows through.

import com.klug.petcam.camera.CameraStreamANE;
import flash.events.Event;
import flash.events.StatusEvent;

var media:CameraStreamANE = CameraStreamANE.instance;
media.addEventListener( StatusEvent.STATUS, onReceive );

// video area between a 80 px header and a 160 px button bar
media.startReceive( "rtmp://your.server:1935/live", "camera-1",
                    0, 80, stage.stageWidth, stage.stageHeight - 240,
                    0, CameraStreamANE.RECEIVE_FIT );
showSpinner();

function onReceive( e:StatusEvent ):void
{
    if ( e.code != "receiveStatus" ) return;
    switch ( e.level )
    {
        case "playing":      break;                    // subscribed, waiting for video
        case "firstFrame":   hideSpinner(); break;     // picture on screen
        case "streamEnded":  showSpinner(); break;     // publisher stopped (iOS, macOS)
        case "disconnected": showOffline(); break;
        default:
            if ( e.level.indexOf( "failed:" ) == 0 ) showOffline();
    }
}

When the publisher stops, iOS and macOS clear the last frame and keep the subscription, so the picture returns by itself when the stream is published again. receivedFrames counts the frames the receiver has handled so far (greater than 0 once video is on screen, 0 when not receiving); you can poll it as a synchronous companion to firstFrame. Calling startReceive() for another stream replaces the running one.

Native views above the stage

The received video and the camera preview are native views. They sit above everything AIR draws: buttons, text, popups and AIR-drawn dialogs inside their rectangle are covered. Plan the layout for this:

• Keep AIR controls outside the video rectangle, or move the rectangle with setReceiveFrame().
• Before you show an AIR dialog or another screen over the video, call hidePreview() and stopReceive() (or move the rectangle away from the dialog). Show them again afterwards.
• On iOS, system alerts presented natively are kept above the extension's views.
• The camera preview is drawn above the received video, so a picture-in-picture self view works.
• On iOS the top edge of the receive view is kept below the status bar and camera housing: if y lies inside the top safe area, the view starts at the safe area and keeps its bottom edge. safeAreaTop returns that inset in stage pixels.

function openSettings():void
{
    media.hidePreview();
    media.stopReceive();
    settingsPanel.visible = true;        // your own AIR panel
}

function closeSettings():void
{
    settingsPanel.visible = false;
    media.startReceive( url, "camera-1", 0, 80, stage.stageWidth, stage.stageHeight - 240 );
}

Rotation, resizing and fullscreen

Native views do not follow the AIR stage on their own. When the stage changes size (device rotation, a foldable opening, a window resize, fullscreen), move them: setReceiveFrame() for the received video, and showPreview() again with the new rectangle for the preview (a visible preview is only moved, not restarted). setReceiveFill(true) switches the received video to crop-to-fill, which suits fullscreen; false returns to aspect-fit.

var fullscreen:Boolean = false;

stage.addEventListener( Event.RESIZE, function( e:Event ):void { layoutVideo(); } );

function layoutVideo():void
{
    var w:int = stage.stageWidth, h:int = stage.stageHeight;
    if ( fullscreen )
        media.setReceiveFrame( 0, 0, w, h );
    else
        media.setReceiveFrame( 0, 80, w, h - 240 );
    media.setReceiveFill( fullscreen );                       // cover in fullscreen

    if ( previewVisible )                                       // your own flag
        media.showPreview( w - 340, h - 400, 320, 240, CameraStreamANE.CAMERA_FRONT );
}

function toggleFullscreen():void
{
    fullscreen = !fullscreen;
    layoutVideo();
}

Camera preview

showPreview() shows the camera picture in a native view at the given rectangle. It works alone (the camera starts in preview-only mode, for example on a setup screen) or beside a live publish (the same camera feeds both). cameraFacing only applies when the preview starts the camera itself; during a publish the preview shows the publishing camera.

• iOS and macOS: a rectangle at least as wide as 4:3 is filled (top and bottom are cropped, like a 16:9 stream); a narrower rectangle shows the whole picture with transparent bars. The front camera is mirrored in the preview only. srcWidth and srcHeight are not used.
• Android: srcWidth and srcHeight set the size of the preview image; pass the shape of your stream (640×480 or 640×360) so the preview matches what receivers see.
• rotationDeg turns the preview view (0, 90, 180, 270 or -90).
• hidePreview() removes the view; the camera stops when nothing else uses it.

// aiming screen before going live: preview only, full camera picture
media.showPreview( 20, 120, stage.stageWidth - 40, (stage.stageWidth - 40) * 3 / 4,
                   CameraStreamANE.CAMERA_BACK, 640, 480, 0 );

function onGoLive():void
{
    media.startPublish( url, "camera-1", 640, 480 );   // preview keeps running
}

function onLeaveSetup():void
{
    media.hidePreview();
}

Switch camera, including USB cameras on the Mac

switchCamera() moves to the next camera while the camera runs (publish or preview). The stream continues; receivers see the new camera after the next keyframe.

• iPhone, iPad, Android: toggles between back and front. With only one camera the call is ignored.
• Mac (native macOS app and iPad app on the Mac): cycles through all cameras: the built-in camera first, then the external USB cameras in a fixed order. A switch takes about a second.

cameraCount tells you whether a switch button makes sense. On Android it counts camera directions (0 to 2) without opening a camera; on iOS and the Mac it counts the cameras the extension can use, including USB cameras. cameraNames lists the cameras as text for logs and support (empty on Android).

switchButton.visible = media.cameraCount > 1;
trace( "cameras: " + media.cameraNames );        // diagnostics only

switchButton.addEventListener( MouseEvent.CLICK, function( e:MouseEvent ):void {
    media.switchCamera();
});

Torch

hasTorch is true when the back camera has a flash unit. Many tablets and all Macs have none, so show a torch button only when it is true. setTorch() switches the light while the camera runs; the extension owns the camera, so this is the only way to control the flash during a stream. On Android the call has an effect only while the camera is running.

var torchOn:Boolean = false;
torchButton.visible = media.hasTorch;

torchButton.addEventListener( MouseEvent.CLICK, function( e:MouseEvent ):void {
    torchOn = !torchOn;
    media.setTorch( torchOn );
});

Zoom

setCameraZoom(step) sets a real camera zoom: steps 0 to 8 map to 1.0x to 2.0x in 12.5 % steps; values outside are clamped. The zoomed picture is what is encoded, so receivers and the preview see the zoom; on iOS snapshots are zoomed too. On iOS the factor is limited to what the camera supports. On Android the step is kept and applied again when the camera restarts. On the native macOS build the call has no effect.

var zoomStep:int = 0;

function zoomIn():void  { zoomStep = Math.min( 8, zoomStep + 1 ); media.setCameraZoom( zoomStep ); }
function zoomOut():void { zoomStep = Math.max( 0, zoomStep - 1 ); media.setCameraZoom( zoomStep ); }

To zoom from the viewing side, send the step to the publishing device with your own messaging (for example a server call) and call setCameraZoom() there.

Snapshots

captureSnapshot(width, height) saves one JPEG from the running camera (publish or preview). It returns false when no camera picture is available; otherwise the result follows as snapshotStatus with saved:<absolute path> or failed:<reason>. On iOS and macOS the camera picture is first cropped to the requested aspect ratio and then scaled.

The file is written to the app's cache folder as klug_snapshot.jpg and is overwritten by the next snapshot. Copy it if you want to keep it.

import flash.filesystem.File;

media.addEventListener( StatusEvent.STATUS, function( e:StatusEvent ):void
{
    if ( e.code != "snapshotStatus" ) return;
    if ( e.level.indexOf( "saved:" ) == 0 )
    {
        var shot:File = new File( e.level.substr( 6 ) );
        var keep:File = File.applicationStorageDirectory.resolvePath( "photo-" + new Date().time + ".jpg" );
        shot.copyTo( keep, true );
        showPhoto( keep.url );                                 // your own function
    }
    else if ( e.level.indexOf( "failed:" ) == 0 )
        trace( "snapshot failed: " + e.level.substr( 7 ) );
});

if ( !media.captureSnapshot( 640, 480 ) )
    trace( "no camera picture yet" );

Bitrate

The bitrate you pass to startPublish() is the target. The extension starts below it and raises the bitrate while the network keeps up; when the upload falls behind, it lowers the bitrate so that no delay builds up. setVideoBitrate(kbps) sets a new target while live. The newBitrate:<value> status reports the current value for logs and displays (Android reports bit/s, iOS and macOS kbit/s).

// user picks a quality while live
function setQuality( level:String ):void
{
    if ( level == "low" )         media.setVideoBitrate( 400 );
    else if ( level == "normal" ) media.setVideoBitrate( 800 );
    else                         media.setVideoBitrate( 1200 );
}

media.addEventListener( StatusEvent.STATUS, function( e:StatusEvent ):void {
    if ( e.code == CameraStreamANE.STATUS_EVENT && e.level.indexOf( "newBitrate:" ) == 0 )
        bitrateLabel.text = e.level.substr( 11 );
});

Two-way voice

Voice is built from three parts: OpusVoiceBridge runs the native audio engine (microphone, speaker, echo cancellation) and turns audio into Opus packets and back; NativeOpusCodec is the Opus codec it uses; Red5OpusTransport carries the packets over your existing NetConnection on two dedicated streams. Each side publishes its own voice stream and plays the other side's.

The loop: incoming packets go to playRemoteOpus(); outgoing audio is collected with readLocalOpus(), which returns one finished 20 ms Opus packet at a time (with the default 48 kHz, 960 frames) or null when no full frame is ready. Drain it in a loop on every frame or on a 20 ms timer.

import com.klug.petcam.audioecho.*;
import flash.events.Event;
import flash.events.NetStatusEvent;
import flash.events.StatusEvent;
import flash.net.NetConnection;
import flash.utils.ByteArray;

var voice:OpusVoiceBridge = new OpusVoiceBridge( new NativeOpusCodec(), 48000, 1, 960 );
var link:Red5OpusTransport;
var nc:NetConnection = new NetConnection();

voice.addEventListener( StatusEvent.STATUS, function( e:StatusEvent ):void {
    if ( e.code == EchoCancellationEvent.ERROR ) trace( "voice error: " + e.level );
});

nc.addEventListener( NetStatusEvent.NET_STATUS, function( e:NetStatusEvent ):void {
    if ( e.info.code == "NetConnection.Connect.Success" ) startVoice();
});
nc.connect( "rtmp://your.server:1935/live" );

function startVoice():void
{
    link = new Red5OpusTransport( nc );
    link.callerID = myDeviceId;                              // your own unique id
    link.addEventListener( OpusPacketEvent.PACKET, function( e:OpusPacketEvent ):void {
        voice.playRemoteOpus( e.packet );                    // decode and play the other side
    });
    link.start( "voice_alice", "voice_bob" );                // on Bob's device: ("voice_bob", "voice_alice")

    voice.setVoiceProcessing( true );                       // two-way echo cancellation, before start()
    if ( !voice.start() ) trace( "voice start failed: " + EchoCancellationANE.instance.lastError );
    EchoCancellationANE.instance.setDuck( false );          // both sides may talk at once
    voice.setSpeakerEnabled( true );                         // hands-free

    addEventListener( Event.ENTER_FRAME, sendVoice );
}

function sendVoice( e:Event ):void
{
    var packet:ByteArray;
    while ( ( packet = voice.readLocalOpus() ) != null )
        link.sendPacket( packet );
}

function stopVoice():void
{
    removeEventListener( Event.ENTER_FRAME, sendVoice );
    voice.stop();
    link.stop();
}

• callerID travels with every packet; packets carrying your own ID are ignored, so give every device its own ID.
• link.ready is true once every requested stream is publishing or playing. Packets sent before that are dropped by the runtime, which is harmless.
• OpusVoiceBridge forwards every status event of the audio engine (started, stopped, error and more; see Events).
• Video and voice are independent: publish and receive video with CameraStreamANE and run the voice streams beside it.

One-way listening and talking

For monitoring, one device sends its sound and the other listens; the listener may talk back now and then. Here no device plays the other side's audio while its own microphone is the main signal, so the echo canceller has nothing to cancel against. Use the one-way settings:

• setVoiceProcessing(false) before start(): on Android loud media-volume playback with the plain microphone; on iOS a lighter voice processing that keeps the room sound faithful. Can also be changed while running on iOS.
• setAntiHowl(true) on the sending device only (iOS): it detects acoustic feedback, for example when the listening phone is in the same room, and turns the microphone down until it stops. Leave it off in two-way calls. No effect on Android.
• setDuck(true) (the default, iOS): the microphone is turned down while far-end audio plays, a half-duplex rule that suits talk-back. Turn it off for two-way calls.
• setCaptureGate(true) only while the local user talks: quiet room noise between words is attenuated. Leave it off for listening, or quiet sounds are cut.

var echo:EchoCancellationANE = EchoCancellationANE.instance;

// sending device (microphone in the room)
voice.setVoiceProcessing( false );
voice.start();
echo.setAntiHowl( true );
link.start( "room_1", "talk_1" );       // send the room, hear talk-back

// listening device: push-to-talk
function onTalkDown():void { voice.setCaptureGate( true );  talking = true;  }
function onTalkUp():void   { voice.setCaptureGate( false ); talking = false; }

function sendVoice( e:Event ):void
{
    var packet:ByteArray;
    while ( ( packet = voice.readLocalOpus() ) != null )
        if ( talking ) link.sendPacket( packet );        // keep draining, send only while talking
}

Gate tuning (iOS): setCaptureGateThreshold() (0 to 32767) is compared with the raw microphone peak before the capture gain; sound at or below it is reduced to setCaptureGateFloor() (a factor, for example 0.12). Measure pauses and speech with getCaptureRms() and set the threshold between them. Too high a threshold cuts real words.

Volume, speaker and delay

• setPlaybackGain(gain): software gain on received audio before playback, 0.0 to 32.0, default 1.0.
• setCaptureGain(gain): software gain on the microphone before Opus encoding, 0.0 to 32.0, default 1.0. It travels with the audio, so the other side hears the change. Raise it for a distant or quiet speaker; too much clips.
• setPlaybackBufferMs(ms): upper bound of the playback buffer, 20 to 2000 ms. When more audio is queued, the oldest audio is dropped, so the delay stays bounded. Larger values ride out longer network gaps; smaller values reduce delay. On iOS and macOS the buffer also adapts to short Wi-Fi stalls by itself and shrinks back when the network is calm.
• setSpeakerEnabled(true): plays through the loudspeaker instead of the earpiece (iOS, Android).

voice.setPlaybackGain( 2.0 );        // louder far end
voice.setCaptureGain( 1.5 );         // slightly louder own voice
voice.setPlaybackBufferMs( 250 );    // allow for Wi-Fi hiccups
voice.setSpeakerEnabled( true );

Microphone level and noise detection

While voice runs, measure the microphone with getCaptureLevel() instead of an AIR Microphone: it returns the peak level since the last call on the same 0 to 100 scale as Microphone.activityLevel, and reading it clears the peak. Poll it at a steady rate. It returns 0 when voice is not running.

var meter:Timer = new Timer( 100 );     // 10 readings per second
meter.addEventListener( TimerEvent.TIMER, function( e:TimerEvent ):void {
    var level:int = EchoCancellationANE.instance.getCaptureLevel();
    levelBar.scaleX = level / 100;
    if ( level > 60 ) onLoudNoise( level );   // your own threshold and handler
});
meter.start();

Weak networks and custom transports

If your transport carries a 16-bit sequence number with each packet, use playRemoteOpusSeq(payload, seq) instead of playRemoteOpus(): gaps in the sequence are filled with Opus forward error correction and packet loss concealment, so a lost packet becomes a short smooth fill instead of a click or dropout.

To carry voice over something other than an RTMP NetConnection, implement IOpusTransport (dispatch OpusPacketEvent.PACKET for incoming packets and StatusEvent with connected, disconnected or error). The bridge does not depend on the transport; it only produces and consumes ByteArray packets. For full control, EchoCancellationANE also exposes the raw PCM and Opus functions.

var echo:EchoCancellationANE = EchoCancellationANE.instance;
echo.start( 48000, 1, 960 );

// echo-cancelled microphone PCM (16-bit little-endian), up to 3840 bytes = 20 ms
var pcm:ByteArray = echo.readCapturePcm16( 3840 );
if ( pcm && pcm.length == 3840 )
{
    var opus:ByteArray = echo.encodeOpus( pcm, 48000, 1, 24000 );
    myTransport.send( opus );                                // your own transport
}

// incoming: decode and play
function onIncoming( opus:ByteArray, seq:int ):void
{
    echo.writePlaybackPcm16( echo.decodeOpusSeq( opus, seq, 48000, 1 ) );
}

Server requirements

Video works with any standard RTMP server: the extension publishes and plays plain H.264 in classic FLV tags. Voice travels as data messages: Red5OpusTransport calls NetStream.send("opusVoicePacket", callerID, packet) on its publish stream, and the server must forward these messages to every subscriber of that stream, which then calls the client method opusVoicePacket(callerID, packet). The Klug streaming server relays them out of the box; with another server, make sure it forwards NetStream.send data messages to subscribers. The voice uses its own pair of streams, separate from the video streams, because some servers relay data messages reliably only on streams without media.

iPad apps on the Mac

On Apple silicon Macs the iOS build of your app can run as an iPad app. The system then reports an iPad model and a low screen density, so the app cannot tell on its own that it runs on a Mac. isiOSAppOnMac is true in exactly this case and false on iPhone, iPad, Android and the native macOS build. Use it to choose a desktop layout, to show a camera switch for USB cameras, or to hide the torch.

if ( media.isiOSAppOnMac )
{
    useDesktopLayout();                          // your own layout
    switchButton.visible = media.cameraCount > 1;  // built-in + USB cameras
    torchButton.visible = false;
}

Camera, preview, receive, snapshots and two-way voice work in this mode, with the built-in camera and external USB cameras; the orientation of external cameras is handled by the extension.

macOS desktop apps

• The native views are placed on the AIR window's content view at the given stage coordinates. After a window resize, call setReceiveFrame() and showPreview() with the new rectangle.
• switchCamera() cycles through all cameras, including USB cameras; it reports progress as informational cameraStatus levels for your log.
• setCameraZoom() has no effect; hasTorch is false.
• A receive-only app never touches the camera, so it does not trigger the camera prompt.

Background and return

The extension does not stop or restart streams by itself when the app leaves the foreground. Mobile systems do not allow camera capture in the background, so plan for it: stop on Event.DEACTIVATE and start again on Event.ACTIVATE. Because a restart publishes the same name again, stop first (see Stop and restart cleanly).

import flash.desktop.NativeApplication;

NativeApplication.nativeApplication.addEventListener( Event.DEACTIVATE, function( e:Event ):void {
    removeEventListener( Event.ENTER_FRAME, sendVoice );
    voice.stop();
    link.stop();
    media.hidePreview();
    media.stopReceive();
    media.stopPublish();
});

NativeApplication.nativeApplication.addEventListener( Event.ACTIVATE, function( e:Event ):void {
    if ( wasLive ) resumeSession();      // your own flag and function: publish, receive, voice
});

Audio interruptions on iOS, such as a phone call, are handled by the extension: when the interruption ends, it restores its audio session and continues (status sessionReasserted).

Error handling

• Calls never throw on platforms without the native part: isSupported is false, Boolean calls return false, other calls do nothing.
• Start calls return a Boolean for the immediate check; everything that happens later arrives as a status event. For video, listen for failed:<reason> and authError:<reason>; for voice, for the code error.
• CameraStreamANE.instance.lastError and EchoCancellationANE.instance.lastError hold the last error text of each part, for logs and support.
• Platform-specific calls that are not available on a platform are ignored there (listed per member in the API reference).

API reference

class CameraStreamANE

Package com.klug.petcam.camera · public final class · extends flash.events.EventDispatcher · singleton. Dispatches StatusEvent.STATUS (see Events).

class OpusVoiceBridge

Package com.klug.petcam.audioecho · public final class · extends flash.events.EventDispatcher. Re-dispatches every StatusEvent of EchoCancellationANE.

class NativeOpusCodec

Package com.klug.petcam.audioecho · public final class · implements IOpusCodec. The native Opus encoder and decoder.

interface IOpusCodec

Package com.klug.petcam.audioecho. Implement it to give OpusVoiceBridge another codec.

class Red5OpusTransport

Package com.klug.petcam.audioecho · public final class · extends flash.events.EventDispatcher · implements IOpusTransport. Carries Opus packets over an RTMP NetConnection on its own publish and play streams. Works with any RTMP server that forwards NetStream.send() data messages.

interface IOpusTransport

Package com.klug.petcam.audioecho · extends flash.events.IEventDispatcher. Implementations dispatch OpusPacketEvent.PACKET for incoming packets and StatusEvent with code connected, disconnected or error.

class OpusPacketEvent

Package com.klug.petcam.audioecho · public final class · extends flash.events.Event.

class EchoCancellationANE

Package com.klug.petcam.audioecho · public final class · extends flash.events.EventDispatcher · singleton. The native audio engine behind OpusVoiceBridge; use it directly for settings the bridge does not expose and for raw PCM and Opus access.

class EchoCancellationEvent

Package com.klug.petcam.audioecho · public final class. Constants for the code of the audio engine's StatusEvents (this is not an event class).

class KlugMediaLicense

Package com.klug.media · public class. Static only.

CameraStreamANE, OpusVoiceBridge, Red5OpusTransport and EchoCancellationANE also inherit addEventListener(), removeEventListener(), hasEventListener() and dispatchEvent() from EventDispatcher.

Events

All native results arrive asynchronously as flash.events.StatusEvent.STATUS; read code for the kind and level for the value. Compare levels with a prefix check (indexOf(...) == 0) where they carry a value after a colon.

On CameraStreamANE.instance

Other codes (webviewStatus, zoomButton, overlayButton, playFileStatus, remoteOpus) belong to functions outside this documentation; ignore them.

On EchoCancellationANE.instance and OpusVoiceBridge

On Red5OpusTransport

KlugMediaLicense sends no events; it returns its result.

Troubleshooting

My buttons or dialog are hidden behind the video.

The received video and the preview are native views above the AIR stage; nothing AIR draws can appear on top of them. Keep controls outside the video rectangle, or hide the preview and stop the receive before you show the dialog. See Native views above the stage.

After rotating the device the video is in the wrong place.

Native views do not follow the stage. Call setReceiveFrame() and showPreview() with the new rectangle in your Event.RESIZE handler.

On iOS the video starts lower than the y I passed.

The receive view keeps its top edge below the status bar and camera housing. If y lies inside the top safe area, the view starts at the safe area and keeps its bottom edge. safeAreaTop gives you the inset.

Publishing the same stream name again fails or shows an old picture.

Call stopPublish() before every new publish of the same name. Some servers keep a dropped stream for a while; a clean stop releases it at once.

startPublish() returns true, but nothing new happens.

An identical call (same URL, name, size, frame rate and camera) while that stream is live is ignored on purpose. Change a parameter, or stop first. Watch the cameraStatus events for the real outcome.

The receiver says playing but shows no picture.

The receiver waits for the first keyframe. Check that the publisher sent connected on the same server and application, with the exact stream name, and keep the keyframe interval short (2 seconds). When the publisher stopped, iOS and macOS send streamEnded and show the picture again on the next publish.

Voice start fails with "microphone permission undetermined".

On iOS the voice start() does not show the prompt itself. Request the microphone permission first with Microphone.permissionManager, then start. On macOS the prompt appears and micPermission follows; call start() again then.

Voice stopped working after I opened an AIR microphone.

A second capture of the microphone in the same process conflicts with the extension's engine; on iOS it reconfigures the audio session. Do not call Microphone.getMicrophone() while voice runs (even reading it creates a capture). Use getCaptureLevel() for levels.

Video works, but the other side hears nothing.

• The server must forward NetStream.send() data messages to subscribers; see Server requirements.
• The stream names must cross: your localStreamName is the other side's remoteStreamName.
• Each device needs its own callerID; packets with your own ID are ignored.
• Keep calling readLocalOpus() in a loop until it returns null, and send every packet.

Feedback howl when both devices are in the same room.

In one-way use the sending device has no far-end signal to cancel. Enable setAntiHowl(true) on the sending device (iOS), use setVoiceProcessing(false) for one-way, and keep the devices apart. In two-way calls use setVoiceProcessing(true).

In a two-way call one side is barely heard.

Turn the duck off for two-way calls: EchoCancellationANE.instance.setDuck(false). With the duck on, the microphone is turned down whenever far-end audio plays, and a constant sound source on the other side keeps it down.

Words are cut while talking.

The capture gate threshold is above the speech level. Lower it with setCaptureGateThreshold(), measured with getCaptureRms(), or leave the gate off.

setCameraZoom() does nothing.

The native macOS build has no camera zoom. On iOS the factor is limited to what the active camera supports.

hasTorch is false, or setTorch() does nothing.

The device has no flash on the back camera (common on tablets, always on Macs). On Android the torch can only be switched while the camera runs.

captureSnapshot() returns false or sends failed:no frame yet.

A snapshot needs a running camera: start a publish or a preview first, and wait for the first frames after starting.

My snapshot file was replaced.

Every snapshot is written to the same cache file. Copy it after saved: if you want to keep it.

RECEIVE_STRETCH looks like RECEIVE_FIT on iOS.

On iOS and macOS the stretch mode is shown aspect-fit. Use setReceiveFill(true) or RECEIVE_CROPFILL to fill the rectangle.

cameraNames is empty.

It is a diagnostic value for iOS and the Mac; on Android it is always empty. Use cameraCount for decisions.

On Android the stream reconnects when I turn the device over.

With streamRotation 0 the picture follows the landscape side the device stands on. On Android the extension restarts the publish for this turn, so you see connecting and connected again.

newBitrate values look very different between platforms.

Android reports bit/s, iOS and macOS kbit/s. Use the value for display and logs only.

Nothing works after EchoCancellationANE.instance.dispose().

dispose() releases the native context of the singleton for good; later calls on it do nothing. Use stop() to pause voice and keep dispose() for the end of the app.

setLicenseKey() returns false.

The key belongs to one application ID: the one you entered at checkout. Check that the app's descriptor <id> matches it and that the key is passed exactly as delivered, without spaces or line breaks.

The app ends at the first camera or microphone access.

The usage descriptions are missing. Add NSCameraUsageDescription and NSMicrophoneUsageDescription to the iOS or macOS InfoAdditions; for signed Mac builds also the camera and audio-input entitlements (see Add the extension).

Android: the crop-to-fill video stays black.

The crop-to-fill mode draws into a TextureView, which needs hardware acceleration. Keep android:hardwareAccelerated="true" on the application.

Changelog