Getting started
Welcome to the Wikitude SDK. This document is designed to help you from your very first steps with the Wikitude SDK all the way through to advanced concepts and examples for developing your augmented reality project.
Recommended Usage of this Documentation
The documentation is arranged in a way to guide you through the various steps in your development process. We recommend following each of the steps outlined below and reading the documentation in the order displayed.
Setup your project
In this section we describe the necessary steps to setup a project in a detailed guide.
View the samples
All of the included samples are complete iOS projects and apps. Browse through this section and get an idea of what the SDK is capable of. The relevant parts of the samples are described in more detail to highlight the applied concepts and patterns. These examples are designed to help you get off to a great start with the Wikitude SDK.
Viewing samples based on vision based augmented reality requires the corresponding reference images. All of them are available directly in the description of the sample or available as a collection on this page, which you can either view on the screen or print.
The Wikitude SDK - Augmented Reality for your own app
The Wikitude SDK is a software library and framework for mobile apps used to create augmented reality experiences.
As opposed to the Wikitude SDK - JavaScript API where you define your AR experiences with JavaScript, you will create your AR experiences with ObjC and OpenGL when using the Wikitude Native SDK.
Architecture of the Wikitude SDK
The image above shows the different components of the Wikitude SDK and possible approaches for creating augmented reality apps. Each of these approaches are based on a certain development environments (IDE) and platforms:
- Computer Vision Engine: The computer vision engine is a core component of the Wikitude SDK and used by all platforms. It is not directly accessible, but wrapped either by the Native API or the JavaScript API.
- Wikitude SDK - Native API: Provides access to the Wikitude computer vision engine natively for Android (Java) and iOS (ObjC). It also can load plugins via the Wikitude Plugins API.
- Wikitude SDK - JavaScript API: Allows to build augmented reality worlds on basis of HTML and JavaScript. It is available for Android and iOS. The JavaScript API provides access to the functionality of the computer vision engine, location based AR, the Plugins API and dedicated rendering functionality.
- Wikitude SDK - Plugins API: An API to connect your own plugins to the Wikitude SDK. (NOTE: Wikitude SDK plugins have nothing to do with the Cordova or Unity Plugin concept.)
- Wikitude SDK - Cordova Plugin: On top of the JavaScript API the Cordova plugin allows to use the Wikitude SDK in combination with Apache Cordova.
- Wikitude SDK - Titanium Module: On top of the JavaScript API the Titanium module allows to use the Wikitude SDK in combination with Titanium.
- Wikitude SDK - Unity Plugin: On top of the Native API the Unity plugin allows to use the Wikitude SDK in combination with Unity.
- Wikitude SDK - Xamarin Component: On top of the JavaScript API the Xamarin component allows to use the Wikitude SDK in combination with Xamarin.
The Wikitude Developer Portal
The Wikitude Developer Section should be your first stop when you have specific development related questions. The portal hosts a very active Developer Community Forum where Wikitude staff members are constantly assisting other developers with helpful tips and advice. A Knowledge Base helps with various questions.
Feedback and Contact
We are always interested in your feedback and suggestions how we can improve this documentation. Please use the contact form on our website or visit us on Google+ or Facebook
Setup Guide iOS Native API
There are only a few steps necessary to add the Wikitude Native SDK to your existing iOS application. This guide will explain them in detail. In general the steps are
- Adding the Wikitude Native SDK .framework to a Xcode project
- Using the main classes to create your augmented reality experiences
Project Setup
Load your Xcode project
The first step is to open an exiting Xcode project. If there is no project already created, do so using the Xcode project configurator.
Adding the Wikitude SDK Framework
To keep things clear, you should copy the Wikitude Native SDK .framework into your Xcode project structure. Having it somewhere else on your machine might lead to confusion when updating to a newer version of the Wikitude Native SDK.
After the .framework was copied, it can be added as 'Embedded Binaries'.
Because the Wikitude Native SDK will fully support Swift, a build setting has to be changed in order to package Swift Standard Libraries into the final application package.
The project is now fully configured to use the Wikitude Native SDK.
Using the Wikitude Native SDK in your Application
After the setup steps are completed, the Wikitude Native SDK is ready to be used within an iOS application.
The Wikitude Native SDK comes with an umbrella header that has to be included in your corresponding source files. The file is called WikitudeNativeSDK.h
and can be imported by calling #import <WikitudeNativeSDK/WikitudeNativeSDK.h>
The main class to work with is WTWikitudeNativeSDK
. It's the central point for creating trackers, to obtain an already configured rendering component or to control camera related settings.
The Wikitude Native SDK can be used in trial mode without any license, but to remove the watermark, a valid license has to be set using the -setLicenseKey
method.
Application Lifecycle
To integrate the Wikitude Native SDK into the application lifecycle, use the -start:completion
and -stop
methods of WTWikitudeNativeSDK
. A good place to call them is e.g. a UIViewControllers
-viewWillAppear:
and -viewWillDisappear:
.
To refine the startup phase, the WTStartupConfiguration
object can be used which is the first arguments in first block parameter of -start:completion:
. The second block parameter can be used to check if the Wikitude Native SDK could actually start with the given configuration on the current device. Use the isRunning
parameter to determine the run state and e.g. create tracker in this block. The following listening demonstrates this:
[self.wikitudeSDK start:nil completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
__weak typeof(self) weakSelf = self;
NSURL *imageTrackerResourceURL = [[NSBundle mainBundle] URLForResource:@"magazine" withExtension:@"wtc" subdirectory:@"Assets"];
self.targetCollectionResource = [self.wikitudeSDK.trackerManager createTargetCollectionResourceFromURL:imageTrackerResourceURL completion:^(BOOL success, NSError * _Nullable error) {
if ( !success )
{
NSLog(@"Failed to load URL resource. Reason: %@", [error localizedDescription]);
}
else
{
weakSelf.imageTracker = [weakSelf.wikitudeSDK.trackerManager createImageTrackerFromTargetCollectionResource:weakSelf.targetCollectionResource delegate:weakSelf configuration:nil];
}
}];
}
}];
Error Handling
The Wikitude Native SDK calls the WTWikitudeNativeSDK
delegate method -wikitudeNativeSDK: didEncounterInternalError:
as soon as an internally inconsistent state was detected and parts of the SDK might stop working as expected. This is especially helpful while developing the application and not so much for end user notifications.
Trackers, Camera and Rendering
How trackers, the camera controls and rendering can be used, will be explained in the example part of this documentation. Each feature will be explained in detail through a real usage example.
Supported iOS Devices
Wikitude SDK (Native API) is running on devices fulfilling the following requirements:
Requirements for other operating systems and platforms are listed in this overview.
Supported Devices
Wikitude SDK is running on devices fulfilling the following requirements:
Sensor-based AR (Geo-AR) | Image recognition and tracking | Instant tracking |
---|---|---|
Android |
||
|
|
|
Android (Native API) |
||
|
|
|
Epson |
||
|
|
|
ODG |
||
|
|
|
iOS (JavaScript API) |
||
|
|
|
iOS (Native API) |
||
|
|
|
ODG |
||
|
|
|
Vuzix |
||
|
|
|
How to obtain a free trial license
The Wikitude SDK requires a valid license key to be able to run properly. An empty or missing license key will block the augmented reality view from showing any meaningful content. You will see a watermark across the screen with the words License Key Missing
. All JavaScript API calls will be ignored and not interpreted.
When downloading the Wikitude SDK you will be forwarded to the license generation page, where a trial license key is automatically generated for you.
Copy the key into your app, which will unlock the trial mode of the Wikitude SDK. The trial mode of the Wikitude SDK contains the full feature set of the Wikitude SDK but will show a Trial
watermark across the screen.
Each trial license key is valid for every application ID on every operating system. You can use the same trial license key in multiple apps.
Where should I enter the license key
iOS Native API
To use the Wikitude iOS SDK with a certain license key, use the method -setLicenseKey:
defined in WTWikitudeNativeSDK.h
and provide a valid license key.
Examples
The following examples should give you an overview of the capabilities offered by the Wikitude SDK. Each sample is capable of running without modifications on all supported platforms.
The Sample App is a fully functional sample project either for Android or iOS. You can easily import it into Android Studio or Xcode and work from there. The user interface is kept very simple and shows a list to select the sample you are interested.
Examples Application
This section describes the SDKExamples
application in detail and highlights the main features and use-cases of the Wikitude SDK.
Please note that the Xcode projects deployment target is set to iOS 8 and the application has to be deployed to a actual iOS device and not the simulator.
Structure
The Native Examples
application is structured as a table view where each cell launches an individual example, describing a specific feature of the SDK.
The application uses an UINavigationViewController
and pushes each example with a dedicated view controller who implements a complete use case. All view controller are grouped together in the Controller
group. Example specific assets are locates in the Assets
group. Rendering related classes that are used by multiple examples can be found in CoreServices
and Views
.
Usage
To run the application, open the Xcode project and click Run
.
Image Recognition
This example shows how to recognize images in the viewfinder and overlay it with images.
For a better understanding, here are some terms that will be used in the following and other section of this documentation related to vision-based augmented reality.
Target: A 2D target image and its associated extracted data that is used by the tracker to recognize an image.
Target Collection: An archive storing a collection of 2D targets that can be recognized by the tracker. A target collection can hold up to 1000 targets. Target collections are stored as
.wtc
filesImageTracker: The tracker analyzes the live camera image and detects the 2D targets stored in its associated target collection. Multiple trackers can be created, however only one tracker can be active for recognition at any given time.
Simple 2D Client Tracking iOS
In this section we will go through the WTSimpleClientTrackerViewController code, which you can find in the example applications folder Native Examples/Controller/ClientTracking
. We will discuss general concepts on how to use the Wikitude Native SDK as we go along, please don't skip this sample even if you are only interested in cloud recognition for example.
Please refer to the Setup Guide section of this documentation to find detailed information on how to integrate the Wikitude Native SDK into a Xcode project.
In this example we will use external rendering, for details on how to setup the rendering and the difference between internal and external rendering please read through the section on rendering.
The WTWikitudeNativeSDK
class is structured to be used in a standard iOS view controller as a model of the MVC pattern. This class is central starting point for anything related to the Wikitude Native SDK. It provides factory methods to create different kinds of tracker and to receive information of those through their delegate objects.
A typical usage would be to have a strong reference to a WTWikitudeNativeSDK
object in a view controller and to define this view controller to be conform to the WTWikitudeNativeSDKDelegate
protocol
@interface WTSimpleClientTrackerViewController () <WTWikitudeNativeSDKDelegate, WTImageTrackerDelegate>
The WTWikitudeNativeSDKDelegate
protocol defines methods that can be used to react on SDK specific events. It is the chosen way to communicate e.g. runtime errors that can not be returned directly from an API call. Such errors might contain information about camera authorization status changes and inconsistent rendering setups.
To receive image tracker information about recognized/tracked or lost targets, this view controller must also conform to the WTImageTrackerDelegate
protocol. All methods in the mentioned protocol are optionals thus only provide information from the SDK but don't require any action from the application.
Additionally this protocol provides information about the loading status of a particular image tracker object.
The next step we will take is create an instance of the WTWikitudeNativeSDK
class. This is the only object that you have to create manually. All other object will be created by factory methods of the WTWikitudeNativeSDK object.
WTWikitudeNativeSDK
manually. All object somehow play together in a well defined manner and that why they are internally controlled by the WTWikitudeNativeSDK
object and its sub components like WTTrackerManger
and WTCaptureDeviceManager
.
A WTWikitudeNativeSDK
object can be instantiated at any time and does not depend on any particular application lifecycle event. To integrate internal processes into the hosting application, -start:completion
and -stop
are used. These two methods should either be called from - viewWillAppear: | - viewWillDisappear:
or -viewDidAppear: | -viewDidDisappear:
.
To refine the startup behaviour of the Wikitude Native SDK, the first block parameter of -start:completion:
contains an already created and setup object of type WTNativeSDKStartupConfiguration
. Public properties of this object can be used to further define how the Wikitude Native SDK should start e.g. its camera. Typical settings from this object are the camera preset which defines in which resolution the device camera should be started or what's the desired target rendering frame rate.
The second block parameter contains a BOOL indicating if the given properties could be taken into account during the final startup phase. If an error occurred, isRunning
is set to NO
and the given NSError
object contains more information about what went wrong.
The second block parameter is also the ideal starting point to create any tracker object from the Wikitude Native SDK if the running state is YES.
Image trackers work with target collection resources, that load target collections represented through a .wtc file (Wikitude Target Collection). The .wtc file can be packed into the application bundle and a file URL retrieved using the NSBundle
method -URLForResource:withExtension:subdirectory:
or uploaded to a server and downloaded from the Wikitude Native SDK.
First, the TargetCollectionResource
is created through the createTargetCollectionResourceFromURL:completion
factory method from WTWikitudeNativeSDK
and when it successfully finished loading the file containing the target collection, an ImageTracker
can be created with it as a parameter.
To get notified when the image tracker finished loading, its delegate method -imageTrackerDidLoadTargets:
is called or -imageTracker:didFailToLoadTargets:
if the .wtc could not be loaded. If the target collection resource finished successfully, but the image tracker failed to load the targets, it means that the file was loaded successfully, but the image tracker was unable to use it, either because it is corrupted, or because it is from an unsupported version, or if the file is not a target collection at all.
[self.wikitudeSDK start:nil completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
__weak typeof(self) weakSelf = self;
NSURL *imageTrackerResourceURL = [[NSBundle mainBundle] URLForResource:@"magazine" withExtension:@"wtc" subdirectory:@"Assets"];
self.targetCollectionResource = [self.wikitudeSDK.trackerManager createTargetCollectionResourceFromURL:imageTrackerResourceURL completion:^(BOOL success, NSError * _Nullable error) {
if ( !success )
{
NSLog(@"Failed to load URL resource. Reason: %@", [error localizedDescription]);
}
else
{
weakSelf.imageTracker = [weakSelf.wikitudeSDK.trackerManager createImageTrackerFromTargetCollectionResource:weakSelf.targetCollectionResource delegate:weakSelf configuration:nil];
}
}];
}
}];
Next we will have a closer look on how to get on tracker specific data. The WTImageTrackerDelegate
provides three methods that will be called when the tracker found, lost or tracked targets.
-imageTracker:didRecognizeImage:
is called if a new target was found in the current camera frame-imageTracker:didTrackImage:
is called when an already known target moved to a new position-imageTracker:didLoseImage:
is called when an already known target could not be found anymore in the latest camera frame
The WTImageTarget object provides information about which target changed and what changed for the target, e.g. its position which would be represented through a OpenGL like model view matrix.
- (void)imageTracker:(nonnull WTImageTracker *)imageTracker didRecognizeTarget:(nonnull WTImageTarget *)recognizedTarget
{
NSLog(@"recognized target '%@'", [recognizedTarget name]);
_isTracking = YES;
}
- (void)imageTracker:(nonnull WTImageTracker *)imageTracker didTrackTarget:(nonnull WTImageTarget *)trackedTarget
{
[self.renderableRectangle setProjectionMatrix:trackedTarget.projection];
[self.renderableRectangle setModelViewMatrix:trackedTarget.modelView];
}
- (void)imageTracker:(nonnull WTImageTracker *)imageTracker didLoseTarget:(nonnull WTImageTarget *)lostTarget
{
NSLog(@"lost target '%@'", [lostTarget name]);
_isTracking = NO;
}
- (void)imageTrackerDidLoadTargets:(nonnull WTImageTracker *)imageTracker
{
NSLog(@"Image tracker loaded");
}
- (void)imageTracker:(nonnull WTImageTracker *)imageTracker didFailToLoadTargets withError:(nonnull NSError *)error
{
NSLog(@"Unable to load image tracker. Reason: %@", [error localizedDescription]);
}
Extended 2D Image Tracking iOS
Based on the above to enable Extended Tracking for a tracker you need to provide a String array which defines which targets should be extended. In the sample we simply set a wildcard *
so that all targets in this tracker are extended.
weakSelf.imageTracker = [weakSelf.wikitudeSDK.trackerManager createImageTrackerFromTargetCollectionResource:weakSelf.targetCollectionResource delegate:weakSelf configuration:^(WTImageTrackerConfiguration *imageTrackerConfiguration) {
imageTrackerConfiguration.extendedTargets = @[@"*"];
}];
Cloud Recognition
This example shows how to recognize images on a cloud server and then overlay it with augmentations utilizing the ImageTracker
and CloudRecognitionService
classes.
For a better understanding, here are some terms that will be used in the following and other sections of this documentation related to vision-based augmented reality.
Target: An image and its associated extracted data that is used to recognize an image.
Target Collection: A group of
targets
that are searched together. Think of it as a directory, which contains all your images you want to search. The Wikitude SDK can work with two different sorts ofTarget Collections
- On-device Target Collection: a static
wtc
file containing the extracted data of your images. Can consist of up to 1,000 images. - Cloud Target Collection: A target collection stored on the Wikitude server. See
Cloud Archive
below. Can consist of up to 50,000 images.
- On-device Target Collection: a static
Cloud Archive: An archive stored on the server that is optimized for cloud-based recognition. It is generated from a
Target Collection
and is used in combination withCloudRecognitionService
.CloudRecognitionService: Instead of analysing and computing the live camera feed directly on the device, the
CloudRecognitionService
will send the image(s) taken by the camera to the Wikitude Cloud Recognition server. The server will then do the hard work of trying to match the image with your targets in the specified cloud archive. Beside the benefit of searching in large image database, using theCloudRecognitionService
has also a positive impact on the general performance in most cases. Especially when using a large target collection and on older devices.
For both Cloud Recognition samples below we will use external rendering. If you don't know what that means please go through the section on rendering before starting here.
The CloudRecognitionService
is able to run in two modes, called on-click and continuous. In On-Click mode a single recognition cycle will be executed, while in continuous mode the recognition will be run repeatedly with a variable interval.
Both view controllers are located in Controller/CloudTracking
.
Regional server endpoints
Before we get started please note that you have to choose which regional-distributed Wikitude server the SDK should contact.
The cloud recognition server region can be selected by calling -setCloudRecognitionServerRegion:
which is a method of WTTrackerManager
. As a parameter, pass a value of WTCloudRecognitionServerRegion
to specify the regional server. Every cloud recognition service that is created afterwards, will contact the specified server.
On-Click Cloud Tracking
The starting point for on-click recognition is WTOnClickCloudRecognition
. In -viewDidLoad
we create a new instance of WTWikitudeNativeSDK
and initialize it with a valid license key.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
[self.wikitudeSDK setLicenseKey:kWTLicenseKey];
Then in -viewDidAppear:
the Wikitude Native SDK is started using the -start:completion:
method. If the SDK could be started, a object of type WTCloudRecognitionService
is created using the WTTrackerManager
factory method -createCloudRecognitionServiceWithClientToken:targetCollectionId:completion
. The returned pointer needs to be retained in order to keep it alive, so assign it to a strong property. In the completion block we need to check if the cloud recognition service was successfully initialized, and if so, we can create an image tracker with it by using the -createImageTrackerFromCloudRecognitionService:delegate:configuration
factory method.
[self.wikitudeSDK start:nil completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
__weak typeof(self) weakSelf = self;
self.cloudRecognitionService = [self.wikitudeSDK.trackerManager createCloudRecognitionServiceWithClientToken:@"b277eeadc6183ab57a83b07682b3ceba" targetCollectionId:@"54e4b9fe6134bb74351b2aa3" completion:^(BOOL success, NSError * error) {
if ( !success )
{
NSLog(@"Cloud recognition service failed to initialize with error: %@", [error localizedDescription]);
}
else
{
NSLog(@"Creating image tracker");
weakSelf.imageTracker = [weakSelf.wikitudeSDK.trackerManager createImageTrackerFromCloudRecognitionService:weakSelf.cloudRecognitionService delegate:weakSelf configuration:nil];
}
}];
}
}];
When an image tracker is initialized with a cloud recognition service, the imageTrackerDidLoadTargets
and imageTracker:didFailToLoadTargets
are not called during initialization, but rather after receiving a valid response from the server and they inform us whether the tracker was able to load and initialize the targets received it received from the server.
- (void)imageTrackerDidLoadTargets:(nonnull WTImageTracker *)imageTracker
{
NSLog(@"Image tracker loaded");
}
- (void)imageTracker:(WTCloudTracker * __nonnull)imageTracker didFailToLoadTargets:(NSError * __nonnull)error
{
NSLog(@"Image tracker failed to load with error: %@", [error localizedDescription]);
}
To start a single recognition, call the -recognize:
method. The first parameter in the block is the response returned from the cloud recognition service. To call the -recognize:errorHandler:
method, a UIButton
is set up to call a specific method once it was touched.
- (IBAction)sendCloudRecognitionRequest:(id)sender
{
if ( self.cloudRecognitionService && self.cloudRecognitionService.initialized )
{
[self.cloudRecognitionService recognize:^(WTCloudRecognitionServiceResponse * _Nullable response, NSError * _Nullable error) {
if ( response )
{
if ( response.recognized )
{
NSLog(@"Recognized target '%@' which has a rating of '%ld'", [response.targetInformations objectForKey:WTCloudRecognitionServiceResponseKey_TargetName], (long)[[response.targetInformations objectForKey:WTCloudRecognitionServiceResponseKey_TargetRating] integerValue]);
NSLog(@"Associated (custom) metadata: %@", response.metadata);
}
else
{
NSLog(@"No target image found in the analysed camera frame.\nKeep Calm \n\t&\nKeep Looking");
}
}
else
{
NSLog(@"Cloud recognition recognition failed. Reason: %@", [error localizedDescription]);
}
}];
}
}
Please note that only a recognize call is made if the cloud recognition service finished initializing.
In case the server communication was successful, the first parameter in the block is a valid object of type WTCloudRecognitionServiceResponse
that represents the server result. This object contains information if a image target was found or not, its name and, if specified, its associated meta data.
In case of an error, the second parameter will contain a valid NSError
object which contains detailed information about why the communication failed.
Since both simple image tracking and this use the WTImageTrackerDelegate
protocol, all tracking related methods to draw a augmentation around the recognized target are available as well.
Continuous Cloud Tracking
On-Click recognition is useful in some particular cases, but more often than not you probably want to use continuous recognition. For continuous cloud recognition we set an interval in which the CloudRecognitionService
automatically calls the recognize function.
To start continuous recognition, simply call the -startContinuousRecognitionWithInterval:interruptionHandler:responseHandler:
cloud recognition service method once it finished initializing.
if ( self.cloudRecognitionService && self.cloudRecognitionService.initialized )
{
__weak typeof(self) weakSelf = self;
[self.cloudRecognitionService startContinuousRecognitionWithInterval:1.5 interruptionHandler:nil responseHandler:^(WTCloudRecognitionServiceResponse *response, NSError * error) {
if (response)
{
NSLog(@"received continuous response...");
if ( response.recognized ) {
NSLog(@"target recognized...");
dispatch_async(dispatch_get_main_queue(), ^{
[weakSelf.cloudRecognitionService stopContinuousRecognition];
[weakSelf.continuousRecognitionButton setTitle:@"Start Continuous Recognition" forState:UIControlStateNormal];
});
}
else
{
NSLog(@"target NOT recognized...");
}
}
else
{
NSLog(@"Cloud Recognition Service error %ld occurred. %@", (long)error.code, [error localizedDescription]);
}
}];
[self.continuousRecognitionButton setTitle:@"Stop Continuous Recognition" forState:UIControlStateNormal];
}
else
{
NSLog(@"Cloud Recognition Service is not ready yet.");
}
As before, this method is called once a UIButton
was touched.
The first parameter defines in which interval new camera frames should be sent to the server for processing. The second parameter is again a response handler with information about the recognized target and behaves the same way as in the case of On-Click recognition.
To stop a continuous recognition session, e.g. once a target was found, simply call the -stopContinuousRecognition
method.
Instant Tracking
The following sections detail the instant tracking feature of the Wikitude Native SDK by introducing a minimal implementation, showcasing the simplicity the Wikitude Native SDK provides.
The example is preceded by an introductory section that introduces the instant tracking algorithm with brevity.
Introduction (1/2)
Instant tracking is an algorithm that, contrary to those previously introduced in the Wikitude SDK, does not aim to recognize a predefined target and start the tracking procedure thereafter, but immediately start tracking in an arbitrary environment. This enables very specific use cases to be implemented.
The algorithm works in two distinct states; the first of which is the initialization state. In this state the user is required to define the origin of the tracking procedure by simply pointing the device and thereby aligning an indicator. Once the alignment is found to be satisfactory by the user (which the users needs to actively confirm), a transition to the tracking state is performed. In this state, the environment is being tracked, which allows for augmentations to be placed within the scene.
The instant tracking algorithm requires another input value to be provided in the initialization state. Specifically, the height of the tracking device is required in order to accurately adjust the scale of augmentations within the scene. To this end, the example features a range input element that allows the height to be set in meters.
Basic Instant Tracking (2/2)
The Instant Tracking example provides a minimal implementation of the instant tracking algorithm. First we have to make some additions to our UIViewController
. In order to use instant tracking, the UIViewController
has to conform to the WTInstantTrackerDelegate
protocol and we will need a WTInstantTracker
and a WTInstantTrackingState
as members.
@interface WTInstantTrackerViewController () <WTWikitudeNativeSDKDelegate, WTInstantTrackerDelegate>
@property (nonatomic, strong) WTInstantTracker *instantTracker;
@property (nonatomic, assign) WTInstantTrackingState trackingState;
Initialize both the WTInstantTracker
and the WTInstantTrackingState
after checking if the WTWikitudeNativeSDK
is running in viewDidAppear:
.
self.instantTracker = [self.wikitudeSDK.trackerManager createInstantTracker:self configuration:nil];
self.trackingState = WTInstantTrackerInitializing;
A WTInstantTracker
can, minimally, be instantiated with just the previously generated tracker instance, although supplying drawables to be rendered in both the initialization state and the tracking state is advisable for any practical use case. Therefore a StrokedRectangle
instance is generated and supplied as well.
self.renderableRectangle = [[StrokedRectangle alloc] init];
self.renderableRectangle.scale = 0.45;
To use our StrokedRectangle
for the initialization of WTInstantTracker
, we implement the instantTracker:didChangeInitializationPose:
callback function of the WTInstantTrackerDelegate
.
- (void)instantTracker:(nonnull WTInstantTracker *)instantTracker didChangeInitializationPose:(nonnull WTInitializationPose *)pose
{
[self.renderableRectangle setProjectionMatrix:pose.projection];
[self.renderableRectangle setModelViewMatrix:pose.modelView];
}
This callback function supplies a WTInstantTracker
and the current WTInitializationPose
, the latter of which can be used to set the projection matrix and model view matrix of the StrokedRectangle
so it is displayed properly during the initialization state.
Next we need a means to transition from one state to the other. For this task we provide the toggleInstantTrackingState:
function which we conveniently call on a button click. self.trackingState
holds the value of the current state.
- (IBAction)toggleInstantTrackingState:(id)sender
{
if ( [[self.instantTrackingButton currentTitle] isEqualToString:@"Start Tracking"] )
{
if ( WTInstantTrackerInitializing == self.trackingState )
{
[self.instantTrackingButton setTitle:@"Start Initialization" forState:UIControlStateNormal];
[self.instantTracker setActiveInstantTrackingState:WTInstantTrackerTracking];
}
}
else
{
if ( WTInstantTrackerTracking == self.trackingState )
{
[self.instantTrackingButton setTitle:@"Start Tracking" forState:UIControlStateNormal];
[self.instantTracker setActiveInstantTrackingState:WTInstantTrackerInitializing];
}
}
}
We now draw an orange rectangle while the instant tracker is initializing, but we want the rectangle to be blue once we're in the tracking state.
- (void)instantTracker:(nonnull WTInstantTracker *)instantTracker didChangeState:(WTInstantTrackingState)newState
{
_trackingState = newState;
if (_trackingState == WTInstantTrackerInitializing) {
[self.renderableRectangle setColor:[UIColor colorWithRed:1.00f green:0.58f blue:0.16f alpha:1.0f]];
} else {
[self.renderableRectangle setColor:[UIColor colorWithRed:0.41f green:0.60f blue:0.76f alpha:1.0f]];
}
}
In the WTInstantTracker
's callback function instantTracker:didChangeState:
we control the color of the StrokedRectangle
for a better visual representation. However this only changes the color of the rectangle and not its position, which is why we need the instantTracker:didTrack:
callback function to update the projection matrix and the model view matrix of the rectangle.
- (void)instantTracker:(nonnull WTInstantTracker *)instantTracker didTrack:(nonnull WTInstantTarget *)target
{
[self.renderableRectangle setProjectionMatrix:target.projection];
[self.renderableRectangle setModelViewMatrix:target.modelView];
}
Next, we provide the updateDeviceHeightAboveGround:
function to set the deviceHeight
property of the WTInstantTracker
and connect it to a UISlider
. While this change is, strictly speaking, not required, we strongly recommend every application to supply the device height accurately by this method or another for the Wikitude SDK to provide an accurate scale.
- (IBAction)updateDeviceHeightAboveGround:(UISlider *)sender
{
[self.instantTracker setDeviceHeightAboveGround:@(sender.value)];
}
Lastly we have to make the height slider invisible when the state changes from initialization to tracking, so set its alpha value correctly in instantTracker:didChangeState:
.
dispatch_async(dispatch_get_main_queue(), ^{
if ( WTInstantTrackerTracking == newState )
{
self.deviceHeightAboveGroundSlider.alpha = 0.0;
}
else
{
self.deviceHeightAboveGroundSlider.alpha = 1.0;
}
});
The example outlined in this section renders an orange rectangle augmentation when in initialization state as its indicator and a corresponding blue rectangle augmentation when in tracking state. While the example is quite trivial, we believe it serves the purpose of familiarizing the reader with the core concepts of instant tracking well.
Rendering
This example shows and explain how rendering works in combination with the Wikitude SDK Native API. There are two methods of rendering available in the Wikitude Native SDK. We call them internal and external rendering. Internal means the OpenGL view is setup by the Wikitude SDK and the SDK user can define custom rendering, that is executed by the Wikitude SDK. On the other hand external rendering means the SDK user sets up the OpenGL view and integrates the Wikitude SDK into this rendering setup.
External Rendering
Most of the examples in the Wikitude Native SDK example application use external rendering, simply because it might be the preferred usage by developers. The example application comes with a simple OpenGL ES 2 view (ExternalEAGLView
) and an also very simple renderer (ExternalRenderer
).
To activate external rendering you need to pass WTRenderingMode_External
to the WTWikitudeNativeSDK
method -initWithRenderingMode:delegate
.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
During the Wikitude Native SDK startup phase, which is initialized with a call to -start:completion:
, several methods from the WTWikitudeNativeSDKDelegate
protocol are called. They are needed to prepare the Wikitude Native SDK for external rendering. All methods in the WTWikitudeNativeSDKDelegate
protocol are marked as optional, but the Wikitude Native SDK will call its -wikitudeNativeSDK:didEncounterInternalError:
delegate method if a required method is not implemented or returned an invalid value. The required methods are:
- -wikitudeNativeSDK:didCreatedExternalUpdateHandler:
- -wikitudeNativeSDK:didCreatedExternalDrawHandler:
- -eaglViewSizeForExternalRenderingInWikitudeNativeSDK:
- -eaglContextForVideoCameraInWikitudeNativeSDK:
- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreatedExternalUpdateHandler:(WTWikitudeUpdateHandler __nonnull)updateHandler
{
self.wikitudeUpdateHandler = updateHandler;
}
- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreatedExternalDrawHandler:(WTWikitudeDrawHandler __nonnull)drawHandler
{
self.wikitudeDrawHandler = drawHandler;
}
- (CGRect)eaglViewSizeForExternalRenderingInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
return self.eaglView.bounds;
}
- (EAGLContext *)eaglContextForVideoCameraInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
if (!_sharedWikitudeEAGLCameraContext )
{
EAGLContext *rendererContext = [self.renderer internalContext];
self.sharedWikitudeEAGLCameraContext = [[EAGLContext alloc] initWithAPI:[rendererContext API] sharegroup:[rendererContext sharegroup]];
}
return self.sharedWikitudeEAGLCameraContext;
}
The first two methods provide block objects, that need to be called every frame from the external rendering system. There purpose is to update internal SDK logic and draw the camera stream in OpenGL.
The last one is used to get to know the current OpenGL view size in order to draw the camera frame fullscreen and with the correct aspect ratio.
External rendering also requires a shared EAGLContext
object that is used by the Wikitude Native SDK to issue camera related OpenGL ES 2 calls. This way custom rendering and Wikitude rendering do not influence each other and less conflicts can occur.
An example of how those block objects can be used is the following:
if ( self.wikitudeUpdateHandler
&&
self.wikitudeDrawHandler )
{
self.wikitudeUpdateHandler();
self.wikitudeDrawHandler();
}
// ...external rendering code...
Such a snippet is typically found somewhere in the external render loop.
Internal Rendering
In case no OpenGL ES 2 rendering is already setup in the hosting Wikitude Native SDK application, the Wikitude Native SDK brings its own set of OpenGL ES 2 compatible classes to do so. These classes are WTEAGLView
and WTRenderer
. Objects of those classes can be retrieved through WTWikitudeNativeSDK
methods. To start the Wikitude Native SDK with internal rendering, pass WTRenderingMode_Internal
to the -initWithRenderingMode:delegate:
method.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_Internal delegate:self];
self.wikitudeRenderer = [self.wikitudeSDK createRenderer];
self.wikitudeEAGLView = [self.wikitudeSDK createEAGLView];
After all required objects are created, view and renderer have to be connected through a call to setRenderer:
.
[self.wikitudeEAGLView setRenderer:self.wikitudeRenderer];
After those two objects are connected, the view has to be added to the view hierarchy of your application. This can either be done using Storyboards or programmatically. The Wikitude Native SDK example application does it in code.
[self.view addSubview:self.wikitudeEAGLView];
[self.wikitudeEAGLView setTranslatesAutoresizingMaskIntoConstraints:NO];
NSDictionary *views = NSDictionaryOfVariableBindings(_wikitudeEAGLView);
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"|[_wikitudeEAGLView]|" options:0 metrics:nil views:views]];
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"V:|[_wikitudeEAGLView]|" options:0 metrics:nil views:views]];
To do some custom OpenGL ES 2 calls from outside the Wikitude Native SDK, custom update and draw block objects can be passed to the Wikitude Native SDK. The corresponding delegate methods are called during the startup phase.
- (WTCustomUpdateHandler)wikitudeNativeSDKNeedsExternalUpdateHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
/* Intentionally returning a nil handler here */
return ^(){};
}
- (WTCustomDrawHandler)wikitudeNativeSDKNeedsExternalDrawHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
return ^(){
if ( _isTracking ) {
[self.renderableRectangle drawInContext:[EAGLContext currentContext]];
}
};
}
In this snippet only custom rendering is done and no update logic.
Plugins API
This guide consists of multiple sections, first we discuss Wikitude SDK Plugins in general, than we talk about platform specifics and how to register a plugin with the Wikitude SDK and then we go through each of the sample plugins included with the Wikitude Example Applications.
- About Wikitude SDK Plugins
- Platform Specifics
- Registering Plugins
- QR & Barcode Plugin
- Face Detection Plugin
About Wikitude SDK Plugins
Technically a plugin is a class, either written in C++ or Java, that is derived from the Wikitude Plugin base class. Beside lifecycle handling and options to enable and disable the plugin, the Plugin class has four main methods that you can override cameraFrameAvailable
which is called each time the camera has a new frame, update
which is called after each image recognition cycle as well as startRender
and endRender
which are called before and after the Wikitude SDK does its rendering.
The most important thing to remember when working with plugins is that they need to have a unique identifier! If the attempt is made to register a plugin with an identifier that is already known to the Wikitude SDK, the register method call will return false.
Plugin Base Class
class Plugin {
public:
Plugin(std::string identifier_);
~Plugin();
string getIdentifier() const; // returns a unique plugin identifier
bool processesColorCameraFrames(); // returns true if the plugins wants to process color frames instead of bw
void setEnabled(bool enabled_);
bool isEnabled();
string callJavaScript(string javaScriptSnippet); // evaluates the given JavaScript snippet in the currently loaded ARchitect World context.
protected:
void initialize(); // called when the plugin is initially added to the Wikitude SDK
void pause(); // called when the Wikitude SDK is paused e.g. the application state changes from active to background
void resume(uint pausedTime_); // called when the Wikitude SDK resumes e.g. from background to active state. pausedTime represents the time in milliseconds that the plugin was not updated.
void destroy(); // called when the plugin is removed from the Wikitude SDK
void cameraFrameAvailable(const Frame&; cameraFrame_); // called each time the camera has a new frame
void update(const vector<RecognizedTarget> recognizedTargets_); // called each time the Wikitude SDK renders a new frame
void startRender(); // called before any Wikitude SDK internal rendering is done
void endRender(); // called right after any Wikitude SDK internal rendering is done
protected:
string _identifier;
bool _enabled;
};
With those methods in place your plugin will be able to read the full camera image for your own purpose, where the YUV image is also processed in wikitude’s computer vision engine.
Information about Recognized Targets
In case you have the wikitude SDK running with ongoing image recognition, the plugin API will populate the RecognizedTarget
in the update
method once an image has been recognized. The plugin can then work with class RecognizedTarget
, which wraps the details of the target image in the camera view. With that you can read out the pose of the target image and use it for your purposes. Additionally the call contains the calculated distance to the recognized target
class RecognizedTarget {
public:
const string& getIdentifier() const; // the identifier of the target. The identifier is defined when the target is added to a target collection
const Mat4& getModelViewMatrix() const; // the model view matrix that defines the transformation of the target in the camera frame (translation, rotation, scale)
const Mat4& getProjectionMatrix() const;
const float getDistanceToCamera() const; // represents the distance from the target to the camera in millimeter
};
Passing values from within the plugin to the JavaScript part of your augmented reality experience is done via the addToJavaScriptQueue()
method of the Plugin class. Using this function will execute any JavaScript code in the context of your augmented reality experience.
Platform Specifics
Thanks to Objective-C++, C++ plugins can be loaded very easily on iOS. Simply create a new C++ file, include and derive from wikitude::sdk::Plugin
and create a std::shared_ptr
from your plugin class. The created shared pointer can then be supplied to the Wikitude SDK specific plugin registration/removal API.
To mark a Objective-C file as Objective-C++, change its file extension from .m to .mm or change the type manually to Objective-C++ using Xcodes Identity and Type inspector.
Registering Plugins
The Wikitude Native SDK for iOS offers three methods to register/remove C++ plugins. All of them are accessible through the WTWikitudeNativeSDK
class.
Registering a C++ Plugin
To register a C++ plugin, call the -registerPlugin:
method and pass a std::shared_ptr
wrapping your C++ plugin pointer. The following snippet shows how to define a property for a shared pointer, its initialization and the registration call.
@property (nonatomic, assign) std::shared_ptr<BarcodePlugin> barcodePlugin;
// ...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self); // arguments are passed to the C++ class constructor
// ...
[self.wikitudeSDK registerPlugin:_barcodePlugin];
Removing a C++ Plugin
To remove a already known C++ plugin, call either the -removePlugin:
or -removeNamedPlugin:
method. The first one takes a shared pointer as argument and tries to find a plugin that matches this plugins identifier. The second one tries to remove a C++ plugin based on a string parameter. The second one is interesting because it allows you to not have a property that holds the shared pointer, but to directly call std::make_shared
when calling -registerPlugin:
. If this is done, the hosting application has no reference to the plugin pointer anymore, but can still remove it using its unique identifier (which has to be known by the hosting application/developer).
[self.architectView removePlugin:_faceDetectionPlugin];
//...
[self.architectView removeNamedPlugin:@"com.wikitude.plugin.face_detection"];
Barcode and QR code reader
This samples shows a full implementation of the popular barcode library ZBar into the Wikitude SDK. As ZBar is licensed under LGPL2.1 this sample can also be used for other projects.
ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code.
The C++ barcode plugin is created when WTBarcodePluginViewController
loads its view object.
- (void)viewDidLoad
{
[super viewDidLoad];
//...
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
//...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self);
}
When WTBarcodePluginViewController
starts the Wikitude Native SDK using the -start:completion:
method, the C++ barcode plugin is registered using the -registerPlugin:
method in the completion block.
- (void)viewDidAppear:(BOOL)animated
{
[super viewDidAppear:animated];
//...
[self.wikitudeSDK start:^(WTStartupConfiguration *startupConfiguration) {
//...
} completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
//...
}
else
{
//...
[self.wikitudeSDK registerPlugin:_barcodePlugin];
}
}];
}
Please note that the plugin is only registered when the Wikitude Native SDK could actually start.
Now let's move to the plugins C++ code. First we'll have a look at the BarcodePlugin.h
file. To create the bar code plugin we derive our BarcodePlugin
class from wikitude::sdk::Plugin
and override initialize
, destroy
, cameraFrameAvailable
and update
. We also declare the following member variables: _worldNeedsUpdate
, _image
, _imageScanner
and _presentingViewController
. The _worldNeedsUpdate
variable will later be used as an indicator if we need to display the recognized barcode, _image
and _imageScanner
are classes from zBar
which we'll use to scan for bar codes and _presentingViewController
will hold a pointer to the WTBarcodePluginViewController
from which we call a method every time we recognize a new barcode or QR code.
#include <zbar.h>
#import <WikitudeNativeSDK/Plugin.h>
#import <WikitudeNativeSDK/Frame.h>
#import <WikitudeNativeSDK/RecognizedTarget.h>
@class WTBarcodePluginViewController;
class BarcodePlugin : public wikitude::sdk::Plugin {
public:
BarcodePlugin(int cameraFrameWidth, int cameraFrameHeight, WTBarcodePluginViewController *presentingViewController);
virtual ~BarcodePlugin();
virtual void initialize();
virtual void destroy();
virtual void cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_);
virtual void update(const std::list<wikitude::sdk::RecognizedTarget>& recognizedTargets_);
protected:
int _worldNeedsUpdate;
zbar::Image _image;
zbar::ImageScanner _imageScanner;
WTBarcodePluginViewController *_presentingViewController;
};
In the constructor we set _worldNeedsUpdate
to zero indicating that there is no update necessary and initialize the zBar::Image
member variable passing its constructor the width and height of the camera frame, the image format of Y800
, set its data pointer to null and the data length to zero.
In the initialize
method we configure the zbar::ImageScanner
by calling setConfig
, enabling all supported bar codes. If you are only interested in one or some particular types of codes, first disabling all bar code types and manually enabling each particular type would be the better idea. That way performance could be greatly improved.
void BarcodePlugin::initialize() {
_imageScanner.set_config(zbar::ZBAR_NONE, zbar::ZBAR_CFG_ENABLE, 1);
}
We react to the destroy
event by setting the current data pointer of the zbar::Image
member to null and length to zero.
void BarcodePlugin::destroy() {
_image.set_data(nullptr, 0);
}
The last but most interesting methods are cameraFrameAvailable
and update
. In the cameraFrameAvailable
method we set the data of our previously initialized zbar::Image
member variable to the frame data we just received and the length of the data to frame width * frame height by calling set_data
. We then start the scanning process by calling the scan
method of our zBar::ImageScanner
passing the zBar::Image
member instance. The zBar::ImageScanner::scan
method returns the number of detected bar codes in the image frame, we save this number in a local variable n
. If n
is not equal to the result of the last frame, which we saved to _worldNeedsUpdate
member variable, we know there was a new bar code detected (meaning there was no bar code in the last frame) or that there was a bar code in the last frame and now there isn't. When that's the case, we do another check if there really was a bar code detected this frame and if there was, we call the presentBarcodeResult
method, passing the code content.
void BarcodePlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
int frameWidth = cameraFrame_.getSize().width;
int frameHeight = cameraFrame_.getSize().height;
_image.set_data(cameraFrame_.getLuminanceData(), frameWidth * frameHeight);
int n = _imageScanner.scan(_image);
if ( n != _worldNeedsUpdate ) {
if ( n ) {
zbar::Image::SymbolIterator symbol = _image.symbol_begin();
[_presentingViewController presentBarCodeResult:[NSString stringWithUTF8String:std::string(symbol->get_data()).c_str()]];
}
}
_worldNeedsUpdate = n;
}
Back in the Objective-C class WTBarcodePluginViewController
, the -presentBarCodeResult:
method is implemented like the following:
- (void)presentBarCodeResult:(NSString *)barcodeScanResult
{
if ( barcodeScanResult ) {
dispatch_async(dispatch_get_main_queue(), ^{
[self.barcodeLabel setText:barcodeScanResult];
});
}
}
Please note that the -presentBarcodeResult:
method is not called from the main thread, meaning that we need to schedule any UI related calls on the main thread. This is done by executing a block asynchronously on the dispatch main queue.
To visualize the detected bar code, we use a NSLabel and set its text property to the detected bar code string.
Face Detection
This samples shows how to add face detection to your Wikitude augmented reality experience using OpenCV.
The Face detection plugin example consists of the C++ classes FaceDetectionPlugin
, FaceDetectionPluginConnector
and the Objective-C class WTFaceDetectionPluginViewController
. We will use OpenCV to detect faces in the current camera frame and OpenGL to render a rectangle around detected faces.
FaceDetectionPluginConnector
is used to connect the FaceDetectionPlugin
class with the WTFaceDetectionPluginViewController
. Since this class mainly exists to ease the implementation of a cross platform plugin, we will not go into any implementation details for this class. We also don't go into any OpenCV or OpenGL details. If one is interested in such topics, the source code is part of the Wikitude Native SDK example application.
WTFaceDetectionPluginViewController
handles the face detection plugin creation and registration exactly as described in the previously bar code example. New for this example is a class method that converts UIDeviceOrientations into an int that is used by the face detection plugin to rotate the camera frame in order to always have faces in the right orientation. This class method is then used every time the device changes its orientation.
__weak typeof(self) weakSelf = self;
[[NSNotificationCenter defaultCenter] addObserverForName:UIDeviceOrientationDidChangeNotification object:[UIDevice currentDevice] queue:[NSOperationQueue mainQueue] usingBlock:^(NSNotification *note) {
weakSelf.faceDetectionPlugin->setFlipFlag( [WTFaceDetectionPluginViewController flipFlagForDeviceOrientation:[[UIDevice currentDevice] orientation]] );
}];
Next we have a look at the FaceDetectionPlugin
class. Again we we will leave out implementation details and focus on how we use the plugin itself. In the cameraFrameAvailable
method we use OpenCV to detect faces in the current camera frame which the Wikitude SDK passes to the plugin. We call the observer which is an instance of the FaceDetectionPluginConnector
to notify the view controller about the result. The plugin base class defines startRender
and endRender
, depending on, if you would like to render on top of or below of all rendering the Wikitude SDK does, you choose one of them, or both to override. To render below all Wikitude rendering we choose startRender
and again call the FaceDetectionPluginConnector
instance which in turn calls the view controller. Since we do not react on the result of the Wikitude SDK image recognition we leave update
blank.
... ctor/dtor ...
void FaceDetectionPlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
wikitude::sdk::Size frameSize = cameraFrame_.getSize();
_scaledCameraFrameWidth = cameraFrame_.getScaledWidth();
_scaledCameraFrameHeight = cameraFrame_.getScaledHeight();
std::memcpy(_grayFrame.data,cameraFrame_.getLuminanceData(), frameSize.height*frameSize.width*sizeof(unsigned char));
//... Control Open CV ...
if ( _result.size() ) {
convertFaceRectToModelViewMatrix(croppedImg, _result.at(0));
_observer->faceDetected(_modelViewMatrix);
} else {
_observer->faceLost();
}
}
void FaceDetectionPlugin::startRender() {
_observer->renderDetectedFaceAugmentation();
}
void FaceDetectionPlugin::update(const std::list<wikitude::sdk::RecognizedTarget> &recognizedTargets_) {
/* Intentionally Left Blank */
}
//... other internally used methods ...
To render a frame around detected faces we created an instance of the StrokedRectangle
class which takes care of rendering a rectangle around faces and all targets of the also active ImageTracker
. When the plugin detects, looses or recalculated the projection matrix it will call the appropriate view controller methods which we use to update the StrokedRectangle
instance. If the Plugin decides it is time to render a frame around a detected face it will call -setFaceIsRecognized:atPosition:
. Since the plugin will only call this method in the startRender
method, we know the current thread is the OpenGL thread and are able to dispatch OpenGL calls.
- (void)setFaceIsRecognized:(BOOL)recognized atPosition:(const float*)modelViewMatrix
{
self.faceDetected = recognized;
if (recognized) {
[self.recognizedFaceRectangle setModelViewMatrix:modelViewMatrix];
}
}
- (void)setFaceAugmentationProjectionMatrix:(const float*)projectionMatrix
{
[self.recognizedFaceRectangle setProjectionMatrix:projectionMatrix];
}
Input Plugins API
This guide provides an introduction into the input plugins API of the Wikitude Native SDK and aims to familiarize the reader with its concepts and constraints. Due to the length and complexity of the corresponding example application source code, it will not be presented in its entirety. Relevant and descriptive source code examples will, however, be provided. Since the Input Plugins API is an extension to the Plugins API, we recommend familiarity with it prior to reading this guide.
- About Wikitude SDK Input Plugins
- Input Plugin Base Class
- Concurrency
- OpenGL Context
- Device Orientation
About Wikitude SDK Input Plugins
The input plugins API provides a means to alter the inputs and outputs of the Wikitude Native SDK. For the input case specifically, custom frame data of arbitrary sources can be supplied as an input to the Wikitude SDK Native API for processing. Complementary, for the output case, the default rendering of the Wikitude SDK Native API can be substituted with more advanced implementations. The custom camera example we provide illustrates both principles in a unified use case. A custom camera stream is supplied as an input and a custom rendering effect is used to augment the rendered output.
Input Plugin Base Class
class InputPlugin: public Plugin {
public:
using InputFrameAvailableNotifier = std::function<int(long frameId, std::shared_ptr<unsigned char> frameData)>;
public:
InputPlugin(std::string identifier_);
virtual ~InputPlugin();
virtual bool requestsInputFrameRendering();
virtual bool requestsInputFrameProcessing();
void notifyNewInputFrame(long frameId_, std::shared_ptr<unsigned char> inputFrame_, bool managedFromOutside_ = false);
InputRenderSettings& getRenderSettings();
InputFrameSettings& getFrameSettings();
virtual void prepareRenderingOfInputFrame(long frameId_);
virtual std::shared_ptr<unsigned char> getPresentableInputFrameData();
virtual void internalError(const std::string& errorMessage);
void setInputFrameAvailableNotifier(InputFrameAvailableNotifier newInputFrameAvailableNotifier);
private:
InputFrameAvailableNotifier _newInputFrameAvailableNotifier;
InputFrameRenderSettings _renderSettings;
InputFrameSettings _frameSettings;
std::unique_ptr<InputFrameBufferController> _inputFrameBufferController;
};
The keen observer will be quick to notice that the InputPlugin
class is derived from the Plugin
class. This allows an InputPlugin
to be handled in the same manner as a regular plugin; therefore Plugin
instantiation and registration are identical and will not be discussed redundantly in this guide. Please refer to the Plugins API guide for a detailed explanation. Instead, we will focus on and demonstrate the use of the newly introduced functions.
Color Space
getFrameSettings.setInputFrameColorSpace(wikitude::sdk::FrameColorSpace::YUV_420_NV21);
The Wikitude Native SDK currently accepts RGB frame data corresponding to the FrameColorSpace::RGB
value, as well as YUV data in 4:2:0 NV21 format corresponding to the FrameColorSpace::YUV_420_NV21
value. The former implies a frame data size of frameWidth * frameHeight * 3
bytes, while the latter implies a frame data size of frameWidth * frameHeight * 3 / 2
bytes.
Field of View
getFrameSettings.setFrameFieldOfView([_camera fieldOfView]);
The setFrameFieldOfView
function parameter needs to be a float
value representing the horizontal field of view angle in degrees of the camera the provided frame was captured with. This value is required for the Wikitude computer vision engine to be able to accurately recognise and track targets within the provided frame. Note that the field of view value may significantly differ from device to device, we therefore recommend querying this value from the frame source directly to ensure representative values. For input image files and input video files this value should be discernible from the corresponding meta data; for an input camera stream this value should be accessible though the corresponding camera API.
Frame Size
getFrameSettings.setInputFrameSize({[_camera videoDimensions].width, [_camera videoDimensions].height});
The setInputFrameSize
function parameter needs to be of type wikitude::sdk::Size<int>
containing the input image width and input image height in pixels. Since this value will be constant for many use cases, you may consider hard-coding it to the appropriate values. Alternatively, as with the previous function, we recommend querying the values from either the input file or the input camera device.
Default Frame Rendering
bool YUVFrameInputPlugin::requestsInputFrameRendering() {
return false;
}
The requestsInputFrameRendering
function can be overridden to provide a boolean value indicating whether the input frame data should be processed by the Wikitude SDK Native API or not. The default implementation returns true, meaning that the frame will be rendered using the internal rendering of the Wikitude SDK Native API to present the frame. Should this function be overridden to return false, the responsibility to present the frame becomes that of the InputPlugin
.
Default Frame Processing
bool YUVFrameInputPlugin::requestsInputFrameProcessing() {
return true;
}
The requestsInputFrameProcessing
function can be overridden to provide a boolean value indicating whether the input frame data should be processed by the Wikitude computer vision engine. The default implementation returns true
, meaning that it will be processed. The plugin will be notified of the recognition results through the update function, as is the case for the regular Plugins API. Should this function to be overridden to return false
, the responsibility to perform the desired algorithms becomes that of the InputPlugin
.
Supplying Image Data
void notifyNewInputFrame(long frameId_, std::shared_ptr<unsigned char> inputFrame_, bool managedFromOutside_ = false);
The notifyNewInputFrame
function needs to be called to pass the actual input frame data to the Wikitude SDK Native API. It requires a unique frame identifier of long
type to be supplied, as well as the frame data itself wrapped into a std::shared_ptr<unsigned char>
. It additionally accepts a boolean value indicating whether the default frame caching should be employed or not. The parameter value defaults to false
, meaning the Wikitude SDK Native API default caching will be used. Should you prefer to supply your own frame caching mechanism, set this value to true
. The default caching mechanism keeps up to 5 recent frames in memory to ensure smooth processing performance. Note that you may need to invoke this method from native code due to the file resource or input stream device only being accessible therein. For this use case we recommend having a look at the custom camera sample application code of the Wikitude Native SDK example application.
Rendering Configuration
InputFrameRenderSettings& getRenderSettings();
The getRenderSettings
function behaviour can be altered to provide a parameterised instance of type InputFrameRenderSettings. The default implementation returns the default constructed _renderSettings
member. Should you like to provide render settings to the internal Wikitude Native SDK that differ from the default constructed values, alter the _renderSettings
accordingly before it is returned.
Frame Caching
virtual void prepareRenderingOfInputFrame(long frameId_);
The prepareRenderingOfInputFrame
function is called whenever a frame has been processed to report its identifier. It is, however, only called if requestsInputFrameRendering
has been overridden to return false
and requestsInputFrameProcessing
has been overridden to return true
. The default implementation of this function releases the frame indicated by the received identifier as well as any older frames from the frame cache. In case of a non-default frame caching mechanism, override this method accordingly. An input parameter of -1
identifies the most recent frame.
Receiving Processed Frame Data
virtual std::shared_ptr<unsigned char> getPresentableInputFrameData();
The getPresentableInputFrameData
function can be called to receive the frame data of the most recently processed frame from the default frame cache. Use this method in case requestsInputFrameRendering
has been overridden to return false
in order to retrieve the current frame data to be rendered. When using a custom frame caching mechanism, this function is obsolete.
Error handling
virtual void internalError(const std::string& errorMessage);
The internalError
function gets called whenever an internal error occurs in the Wikitude Native SDK that was not directly related to input plugins. The input parameter provides a description of the error that occurred.
Internal Use Only
void setInputFrameAvailableNotifier(InputFrameAvailableNotifier newInputFrameAvailableNotifier);
The setInputFrameAvailableNotifier
function is called internally and should not be called anywhere else.
Concurrency
When implementing an InputPlugin
, one needs to be aware that its callback functions are invoked concurrently by the Wikitude SDK Native API. It is therefore necessary to protect against race conditions accordingly. We will present two recommended measures to do so: atomic operations and mutual exclusion.
In order to fully utilize the capabilities of the Input Plugin API one must gather data from several asynchronously called member functions, store them, potentially as member variables, and subsequently use them collectively. These operations might be vulnerable to race conditions.
An example snippet from the custom camera example application code:
void YUVFrameInputPlugin::surfaceChanged(wikitude::sdk::Size<int> renderSurfaceSize_, wikitude::sdk::Size<float> cameraSurfaceScaling_, wikitude::sdk::DeviceOrientation deviceOrientation_) {
// some orientation handling code here
_surfaceInitialized.store(true);
}
void YUVFrameInputPlugin::startRender() {
// some early exit code here
render();
}
void YUVFrameInputPlugin::render() {
// some early exit code here
if (!_surfaceInitialized.load()) {
return;
}
// lots of OpenGL code here
}
#include <atomic>
std::atomic_bool _surfaceInitialized;
The surfaceChanged
function and the startRender
function are invoked concurrently. We rely on a boolean value inside the render function that is set from the surfaceChanged
function, yielding a race condition should boolean reads and writes be non-atomic. In such cases, involving intrinsic data types for which atomic operations are provided by the C++ standard library, we recommend their use. These std::atomics
can either be set and read intuitively through the corresponding operators or though the load
and store
functions.
An alternative snippet for which atomic operations are not available:
void YUVFrameInputPlugin::update(const std::list<wikitude::sdk::RecognizedTarget>& recognizedTargets_) {
// platform specific intialization code here
{ // mutex auto release scope
std::lock_guard<std::mutex> lock(_currentlyRecognizedTargetsMutex);
_currentlyRecognizedTargets = std::list<wikitude::sdk::RecognizedTarget>(recognizedTargets_);
}
}
void YUVFrameInputPlugin::startRender() {
// some early exit code here
render();
}
void YUVFrameInputPlugin::render() {
// early returns and lots of OpenGL code here
{ // mutex auto release scope
std::unique_lock<std::mutex> lock(_currentlyRecognizedTargetsMutex);
if (!_currentlyRecognizedTargets.empty()) {
const wikitude::sdk::RecognizedTarget targetToDraw = _currentlyRecognizedTargets.front();
// early unlock to minimize locking duration
lock.unlock();
// lots of OpenGL code here
}
}
}
The update
function and the startRender
function are invoked concurrently. We, again, rely on data being set from an asynchronous function within our render function. Contrary to the previous case though, an object of type std::list
cannot be set atomically using std::atomics
. Therefore we employ a std::mutex
as a locking mechanism to ensure atomicity. As depicted by the code snippet, we encourage the use of RAII style mutex locking using std::lock_guard
and std::unique_lock
to ensure proper mutex release.
OpenGL Context
Another important issue to be aware of is the availability of a valid OpenGL context during plugin run-time. We guarantee such a valid context to be available during the execution of the startRender
, endRender
, pause
and resume
functions. The former two functions should contain all of the rendering related function calls, while the latter should be used to release and acquire OpenGL related resources as the OpenGL context is likely to be destroyed upon pausing the application and recreated upon resuming the application. Therefore all the previously acquired OpenGL handles are no longer valid and need to be reacquired.
A code snipped from the custom camera example:
void YUVFrameInputPlugin::pause() {
releaseFramebufferObject();
releaseFrameTextures();
releaseVertexBuffers();
releaseShaderProgram();
_renderingInitialized.store(false);
// some additional code here
}
void YUVFrameInputPlugin::startRender() {
if (!_renderingInitialized.load()) {
_renderingInitialized.store(setupRendering());
}
render();
}
We release all the OpenGL resources we previously created and atomically set the _renderingInitialized
flag to false
, causing the rendering environment to be reinitialised during the next execution of the render loop.
Device Orientation
Lastly, we will demonstrate the rendering of an input frame from within an InputPlugin
using OpenGL with device orientations taken into account. While there are alternative ways to achieve the correctly oriented frame renderings, we recommend applying the required transformations as matrices within a custom vertex shader.
We propose the following code in our custom camera example application to compose a matrix that is to be applied to a fullscreen quad:
void YUVFrameInputPlugin::surfaceChanged(wikitude::sdk::Size<int> renderSurfaceSize_, wikitude::sdk::Size<float> cameraSurfaceScaling_, wikitude::sdk::DeviceOrientation deviceOrientation_) {
wikitude::sdk::Matrix4 scaleMatrix;
scaleMatrix.scale(cameraSurfaceScaling_.width, cameraSurfaceScaling_.height, 1.0f);
switch (deviceOrientation_)
{
case wikitude::sdk::DeviceOrientation::DeviceOrientationPortrait:
{
wikitude::sdk::Matrix4 rotationToPortrait;
rotationToPortrait.rotateZ(270.0f);
_orientationMatrix = rotationToPortrait;
break;
}
case wikitude::sdk::DeviceOrientation::DeviceOrientationPortraitUpsideDown:
{
wikitude::sdk::Matrix4 rotationToUpsideDown;
rotationToUpsideDown.rotateZ(90.0f);
_orientationMatrix = rotationToUpsideDown;
break;
}
case wikitude::sdk::DeviceOrientation::DeviceOrientationLandscapeLeft:
{
wikitude::sdk::Matrix4 rotationToLandscapeLeft;
rotationToLandscapeLeft.rotateZ(180.0f);
_orientationMatrix = rotationToLandscapeLeft;
break;
}
case wikitude::sdk::DeviceOrientation::DeviceOrientationLandscapeRight:
{
_orientationMatrix.identity();
break;
}
}
_modelMatrix = scaleMatrix * _orientationMatrix;
// some synchronization code here
}
attribute vec3 vPosition;
attribute vec2 vTexCoords;
varying mediump vec2 fTexCoords;
uniform mat4 uModelMatrix;
void main(void)
{
gl_Position = uModelMatrix * vec4(vPosition, 1.0);
fTexCoords = vTexCoords;
}";
struct Vertex
{
GLfloat position[3];
GLfloat texCoord[2];
};
Vertex _vertices[4];
_vertices[0] = (Vertex){{1.0f, -1.0f, 0}, {1.0f, 0.0f}};
_vertices[1] = (Vertex){{1.0f, 1.0f, 0}, {1.0f, 1.0f}};
_vertices[2] = (Vertex){{-1.0f, 1.0f, 0}, {0.0f, 1.0f}};
_vertices[3] = (Vertex){{-1.0f, -1.0f, 0}, {0.0f, 0.0f}};
The matrix composed within the surfaceChanged
function is supplied to the vertex shader as a uniform
parameter and subsequently used to transform the input vertices. Be aware though, that an additional matrix may be required depending on whether you previously rendered to a FBO. If so, the following amendment to the surfaceChanged
function should correct the flipped Y-axis resulting from this process:
wikitude::sdk::Matrix4 scaleMatrix;
_fboCorrectionMatrix.scale(1.0f, -1.0f, 1.0f);
// same device orientation code here as depicted above
_modelMatrix = scaleMatrix * _orientationMatrix * _fboCorrectionMatrix;
For a complete implementation of an input plugin for a specific and advanced use case, we strongly recommend looking into the custom camera example application source code. Additionally, the custom camera sample source code is an excellent starting point to build your own implementation from.
Hardware Control
Camera Settings
The first part of the series will present a general overview of what is possible with the Wikitude SDK.
Camera Position
The camera position defines if the front, back or default camera should be started.
This can be set by using WTNativeStartupConfiguration.captureDevicePosition
or WTCaptureDeviceManager.activeCaptureDevicePosition
.
Camera Focus Mode
The camera focus mode defines which focus mode is used internally by the camera.
It can be set by using WTNativeStartupConfiguration.captureDeviceFocusMode
or WTCaptureDeviceManager.focusMode
.
The camera focus mode can be one of the following modes:
- Continuous: Is the default if the device supports it, in this mode the camera will try to refocus automatically when necessary.
- Once: Can force the camera to refocus once on the current view. If you need to change focus you can set this value again. Each time the value
ONCE
is set, the camera tries to focus on the current scene again - OFF: Is disabling auto focus and will set the focus to the specified manual focus distance.
Manual Camera Focus
The focus distance ranges from 0 to 1 where 0 is closest focus possible and 1 is at infinity focus.
It can be set by using WTNativeStartupConfiguration.captureDeviceFocusDistance
or WTCaptureDeviceManager.focusDistance
.
Camera Resolution
It can be set by using WTNativeStartupConfiguration.captureDeviceResolution
and not be changed during runtime. The recommended values are the following:
Recommendations | Geo | Image Tracking | Extended Tracking | Cloud Recognition | Instant Tracking |
---|---|---|---|---|---|
WTCaptureDeviceResolution |
AUTO |
AUTO |
SD_640x480 |
SD_640x480 |
SD_640x480 |
This setting will only be used on armv8 devices, all other architectures will default to SD_640x480
.
Camera Frame Rate
The camera frame rate can be set to either 30 or 60 fps. If 60 fps is selected the camera resolution may be changed for the 60 fps to be possible.
The first supported combination is used by the SDK:
- FullHD 60 fps
- HD 60 fps
- SD 60 fps
- FullHD 30 fps
- HD 30 fps
- SD 30 fps
It can be set by using WTNativeStartupConfiguration.targetFrameRate
.
Camera Zoom
Digital zoom can be controlled by using WTCaptureDeviceManager.zoomLevel
.
To get the maximum zoom level use WTCaptureDeviceManager.maxZoomLevel
.
Flashlight
The flashlight can be turned on and off by using WTCaptureDeviceManager.torchMode
.
Camera Controls
The WTCaptureDeviceManager
allows you to change capture device specific settings during an active capture session. It lets you change e.g.
* the capture device position
* the capture device focus mode
* the capture device focus distance
* the capture device zoom level
A valid pointer to a WTCaptureDeviceManager
object can be retrieved from the WTWikitudeNativeSDK
method -captureDeviceManager
. Please note that this pointer is only valid if the native SDK is running. If the SDK is currently stopped, changes for the capture device are done using the WTStartupConfiguration
object that is passed as argument to the startupHandler when calling -start:completion:
.
The sample uses a UIPickerView
to let the user select between different camera specific values. This UIPickerView
is added to the view hierarchy in the main storyboards camera scene. By default, the picker view is placed behind the OpenGL ES 2 view, so that it is not visible. To show the picker view, the bottom OpenGL ES 2 view constraint is modified and animated, if a button is touched, to unveil the picker view. A standard iOS SDK UIView animation is used to do so. Please refer to the WTCameraControlsViewController
-handleCameraControlsPresentation
method.
The connection between the UIPickerView
and WTCaptureDeviceManager
is done by a custom class called WTCameraControlsModel
and its delegate object (Which is in this case WTCameraControlsViewController
). This custom class WTCameraControlsModel
implements the UIPickerView
data source and delegate protocol and uses its custom delegate object to retrieve current camera specific values or exposes newly selected values from the picker view.
Implementation details of WTCameraControlsModel
are not discussed in this guide as it only handles UIKit related functionality. The following snippets shows the previously mentioned object connection. All following snippets can be found in the WTCameraControlsViewController
class.
self.cameraControlsPickerModel = [[WTCameraControlsModel alloc] init];
self.cameraControlsPickerModel.delegate = self;
self.cameraControlPicker.dataSource = _cameraControlsPickerModel;
self.cameraControlPicker.delegate = _cameraControlsPickerModel;
The camera controls model knows internally about all possible camera settings and uses these information to load the picker view. Once the picker is loaded and the user selects a different value, the camera controls model delegate is used to propagate the value change. Since the camera controls view controller implements this delegate protocol and also holds a reference to the Wikitude Native SDK, which by itself provides access to the capture device manager), the view controller sets the appropriate capture device value when this camera controls model delegate is called. The following snippet demonstrates this for the capture device position:
- (void)cameraControlsModel:(WTCameraControlsModel *)cameraControlsModel didSelectCameraPosition:(AVCaptureDevicePosition)activeCameraPosition
{
if ( [self.wikitudeSDK captureDeviceManager] ) {
[[self.wikitudeSDK captureDeviceManager] setActiveCaptureDevicePosition:activeCameraPosition];
}
}
Please note that there is a nil check when [self.wikitudeSDK captureDeviceManager]
is called. This is because the Wikitude Native SDK could be paused when the picker changes one of its values.
Every time the picker becomes visible again, the camera controls model asks through its delegate object which capture device settings are currently active. The following snippet again demonstrates this for the currently active capture device position.
- (AVCaptureDevicePosition)currentlyActiveCameraPositionForCameraControlsModel:(WTCameraControlsModel *)cameraControlsModel
{
return [[self.wikitudeSDK captureDeviceManager] activeCaptureDevicePosition];
}
Wikitude 2D Tracking
In this section of the documentation we are covering tools shipped with the Wikitude SDK, which help you in your development.
Target Management
This guide gives you an overview of how to create a target collection that you can use to detect and track images within your augmented reality experience.
In general the conversion can be done via four different tools:
- Wikitude Studio Manager: A browser based tool to convert your images to a wtc file. You can find the tool under: https://targetmanager.wikitude.com. You need your free developer account to log-in. This tool is described in more detail further below.
- WTC Editor within Unity Editor: The Wikitude Unity plugin installs a WTC Editor as extension of the Unity Editor. Unity developers can manage all their targets and target collections directly within Unity Editor. Please consult the Unity documentation on more details.
- RESTful API: The Cloud Recognition Manager API provides a RESTful API to upload target images and convert them to wtc files. Read more details in the section about Cloud Recognition API
- Targets Enterprise Script: A binary shell script available for Mac OS X and Linux converting images to target collections. Pleases contact Wikitude Sales team for technical requirements and pricing.
The following images describes the relationship between the above mentioned methods and the Wikitude Cloud Recognition Service, which is not scope of this documentation.
Wikitude Studio Manager
Add a new project
- Open https://targetmanager.wikitude.com and login with your Wikitude developer account
- Add a new project to your project collection
Add target images
- Enter an existing project
- Add new target images to the project either by clicking on
Add Targets
or drag & drop them on the empty area. Supported file formats include PNG and JPEG. If you are using PNG images, please make sure that it does not contain any transparent pixels, only solid coloured images are supported.
- When uploading a target the file name is used as
target name
. It identifies a target in your experience. If thetarget name
is not completely visible, hover over it to reveal the full name or double click the target to enter edit-mode.
If you add your own target images you need the target name to set them in
AR.ImageTrackable
.Star Rating
- 0 stars: Not suitable for tracking. This target image cannot be tracked because it lacks textured features with high local contrast. Please consider choosing another target image.
- 1 star: Limited tracking ability. This target image provides basic tracking performance in good lightning conditions. Please consider improving the image
- 2 stars: Good tracking ability. This target image will track well in most conditions.
- 3 stars: Very good tracking ability. This target image will track very well in most conditions.
General advice for reference images
- Good image characteristics:
- Diversely textured image with high local contrast
- Bad image characteristics:
- Large areas with solid color or smooth color transitions
- Repetitive patterns
- Logos, signs
Create a WTC file
- AR.TargetCollectionResource WTC (Wikitude Target Collection) file which contains all information of the targets that should be recognized. Enter the project you need the file for and click the WTC icon in the toolbar.
- Select the Wikitude SDK version you're using and click Generate to trigger the creation of the WTC file. You will be notified via e-Mai once the file is available for download.
Use project's WTC file in your app
Look at one of the client recognition examples or refer to the relevant reference for instructions on how to use the created target collection for augmentations in your ARchitect Worlds.
Cloud Recognition
Any existing project may also be published to the Cloud to make it accessible for [AR.CloudRecognitionService
] (../Reference/JavaScript%20API/classes/CloudRecognitionService.html).
Click the Cloud icon in the toolbar for more details.
Cloud Recognition is available for free in your testing process but you must purchase a license for productive use. Learn more
Once a project is published it is accessible via Wiktiude SDK using 'Client Token' and 'Target Collection ID' (compare AR.CloudRecognitionService
)
Hints
You may unpublish a project at any time but be aware that this action has immediate effect on your application(s) making use of the credentials.
Metadata in the 'Edit Target' dialog is solely relevant for Cloud Recognition whereat Physical Height is only relevant for distanceToTarget feature.
Leave Physical Height empty if you do not use the distanceToTarget feature of
AR.ImageTrackable
.The Metadata field is very useful. It allows you to attach JSON data to a target. That way you can define any kind of additional data and react on it dynamically in the SDK to e.g. let a button refer to a details page which is defined in the Metadata JSON.
Targets Versioning
Over time the format and the capabilities of Wikitudes target collection data format have changed. The following table summarizes which version of the Wikitude SDK can handle which wtc version. Wikitude Studio Manager is capable of producing wtc files compatible with all mentioned versions. If you need to re-create a target collection go there and select the appropriate SDK version.
Wikitude SDK 3.x | Wikitude SDK 4.0 | Wikitude SDK 4.1 | Wikitude SDK 5.0 | Wikitude SDK 5.1-5.3 | Wikitude SDK 6.0 | |
---|---|---|---|---|---|---|
wtc 3.x | supported | supported | supported | not supported | not supported | not supported |
wtc 4.0 | not supported | supported | supported | not supported | not supported | not supported |
wtc 4.1 | not supported | not supported | supported | supported | supported | supported |
wtc 5.0 | not supported | not supported | not supported | not supported | not supported | supported |
Best practice for target images
This guide gives you an overview of how to create a target collection that you can use to detect and track images within your ARchitect World.
Summary
Preferred images have:
- between 500 to 1000 pixels in each dimension
- Rich contrast
- Evenly distributed textured areas
- Many corner like structures
Unsuitable images have:
- Smaller dimensions than 500 pixels
- Larger than 1000 pixels as they do not provide more accurate results
- Large amounts of text
- Many repetitive patterns
- Large single-colored areas
- Color contrast only e.g. green to red edge), because all images are processed as grayscale images
Optimal Image Dimensions
- Optimal images are sized between 500 and 1000 pixels in each dimension
- Small images do not contain enough graphical information to extract so called feature points. The uniqueness, amount and distribution of features points are the key indicators for good detection and tracking quality
- Larger images do not improve the tracking quality
Low contrast images
- Images with high local contrast and large amount of rich textured areas is best suited for reliable detection and tracking
- Color contrast only (i.e. green to red edge) appears as high contrast to the human eye but is not discriminative to computer vision algorithms as they are operating on grayscale images Tip: For low contrast images, try to increase the contrast of your target image with an image editing tool like Gimp or PhotoShop to improve detection and tracking quality
Distribution of textured areas
- Images with evenly distributed textured areas are good candidates for reliable detection and tracking
- This might be the hardest part to be in control of and often can’t be changed. Tip: Try to crop the most prominent part of your image and use only this as target image.
Images with whitespace
- Single-colored areas or smooth color transitions often found in backgrounds do not exhibit graphical information suitable for detection and tracking. Tip: Try to crop the most prominent part of your image and use only this as target image.
Vector-based graphics
- Logos and vector-based graphics usually consist of very few areas with high local contrast and textured structures and are therefore hard to detect and track.
Tip: Try to add additional elements to the graphic like your logotype or any other specific elements, which can go along with your graphic.
Images with a lot of text
- Images consisting primarily of large areas of text are hard to detect and track.
Tip: Try to have at least some graphical material and images next to your text for your target image.
Repetitive patterns
- Repetitive patterns exhibit the same graphical information information at each feature point and therefore cannot be localized reliably
- Images with slightly irregular structures can convey a similar information to the target audience while providing enough unique feature points to be detected (second image)
Tip: Try a different selection of your image including non pattern parts or use images with irregular patterns
Target Images
All samples
Click here to download all target images
Wikitude Cloud Recognition
The Wikitude Cloud Recognition service is a cloud-based service provided by Wikitude, which recognizes images sent from Android and iOS apps using the Wikitude SDK. The recognized images are then tracked in the live camera feed and can be used for augmented reality experiences.
This documentation focuses on the RESTful API called Manager API, which is used to interact on a backend level with the Cloud Recognition service.
General Definitions
Target: An image and its associated extracted data that is used to recognize an image.
Target Collection: A group of
targets
that are searched together. Think of it as a directory, which contains all your images you want to search. The Wikitude SDK can work with two different sorts ofTargetCollections
- On-device Target Collection: a static
wtc
file containing the extracted data of your images. Can consist of up to 1,000 images. - Cloud Target Collection: A target collection stored on the Wikitude server. See
Cloud Archive
below.
- On-device Target Collection: a static
Cloud Archive: An archive stored on the server that is optimized for cloud-based recognition. It is generated from a
TargetCollection
and is used in combination with the Wikitude SDKAR.CloudRecognitionService
.
Manager API: A RESTful web API allowing developers to interact with the Cloud Recognition server for managing Targets
, TargetCollections
and Cloud Archives
. Only you as a developer uses this API. None of your users of your app will interact with this API.
Client API: The Client API is the interface between the Wikitude SDK and the Cloud Recognition Service. The API itself is encapsulated in the Wikitude SDK class AR.CloudRecognitionService
and not directly accessible. Calls on the client API are called Scans
.
Region: Wikitude is providing several hosting locations for its Cloud Recognition services to cut down unwanted network latency. As a developer you need to choose on which Region
you and your customers want to operate.
Getting Started with the Cloud Recognition Service
Regional availability of Wikitude Cloud Recognition Service
As as a developer using Wikitude Cloud Recognition Service you need to choose which server location you want to use for your projects. Wikitude operates several servers running Wikitude Cloud Recognition Service in different locations world-wide.
As the region servers are separated, content which is stored on one region server is only available on this particular server. Content is not synced across regions. You can choose from the following Regions
Americas
China
Europe
The servers for each region have separate dedicated domain names and therefore different configurations.
Region | Target Manager | Manager API | SDK Setting |
---|---|---|---|
Americas | targetmanager‑us.wikitude.com | https://api-us.wikitude.com | Americas |
China | targetmanager‑cn.wikitude.com | https://api-cn.wikitude.com | China |
Europe | targetmanager.wikitude.com | https://api.wikitude.com or https://api-eu.wikitude.com | Europe |
Preparation
First Steps and General Usage
- Get familiar with the Manager API calls in the API Reference.
- Create a first Target Collection using the
Create Target Collection
endpoint and note down the ID of the Target Collection - Create targets using the
Create Target
endpoint for that particular Target Collection - Important:
Generate a Cloud Archive
for your Target Collection - Go to the Wikitude SDK and create an Android or iOS project
- Use the Client API token to authenticate your Android or iOS project
- Use your Target Collection ID to recognize images
For more information on the available endpoints and how to work with the Manager API see the workflow section.
Instead of creating a TargetCollection
, adding one or more Targets
, and generating a Cloud Archive
by calling the REST API, the Wikitude Targetmanager can be used to perform these steps (1-4) in the browser alternatively.
In case you would like to immediately test the API calls we recommend the tool Postman. It helps you to quickly construct the requests and analyze the responses.
Authentication
The Cloud Recognition Service knows two authentication tokens, that you need in order to work with the service
Manager API token You need this token to authenticate yourself against the RESTful Manager API. The Manager API is used to create, add and delete targets and target collections. The token identifies your developer account. Calls to the Manager API do not count towards your quota limits.
Client API token You need this token to authenticate calls from the Wikitude SDK to the Cloud Recognition services. It again authenticates calls as legitimate. The token is bound to your developer account. Calls from the Wikitude SDK to the service with a wrong or missing token can not access your target collections.
Authentication on the Manager API
The Manager API token must be added to each call towards the Wikitude Cloud Recognition Manager API. The token authenticates the user account that is using the API.
Authentication on the Client API
The Client API token must be added to your app project using the Wikitude SDK. This token is needed additionally beside the SDK license key when working with the AR.CloudRecognitionService
class.
Quota and Limits
General Upload Limit
The Wikitude Cloud Recognition will not accept images bigger than 1024kB (1 MB). Trying to upload images exceeding this file size will result in a HTTP status code 400
together with an error message FILE_SIZE_LIMIT_EXCEED
.
Limits for the Wikitude Cloud Recognition Service
There are two main limitations for the Wikitude Cloud Recognition service that you need to be aware of:
Targets Your token has a certain number of targets that you can upload and store on the cloud service under your developer account. The limit is always counted for your entire developer account and not for a single target collection. The service is not counting single uploads, but how many targets are currently stored in target collections under your account.
Scans Scans are in effect calls from the Wikitude SDK via the Client API to the Cloud Recognition servers. All commercial license come with an allowance of 1,000,000 scans per month per developer account. Note: When using Continuous Search mode multiple calls are made to the server.
Maximum Number of Targets in a Target Collection
A target collection can't exceed 50,000 targets.
Free Trial License for Cloud Recognition
Wikitude provides a trial token for each developer account to try out the Cloud Recognition for free. This trial token has set a quota limit that allows developers to try and test the functionality of the service. Limitations for trial accounts
- Targets: 50,000
- Scans: 1,000 per month
To get your trial token for the REST, please visit the License page. The trial token is directly integrated into the Target Manager frontend.
Commercial Licenses
For production systems, we offer commercial licenses with various quota limits for purchase.
Product | Targets | Scans |
---|---|---|
Cloud Recognition 1000 | 1,000 | 1,000,000 |
Cloud Recognition 10000 | 10,000 | 1,000,000 |
Cloud Recognition 25000 | 25,000 | 1,000,000 |
Cloud Recognition 50000 | 50,000 | 1,000,000 |
Cloud Recognition 100000 (*) | 100,000 | 1,000,000 |
(*) Maximum number of targets per target collection can't exceed 50,000
Your first Target Collections
Target Collections are central to working with Cloud Recognition service. They keep all your target images and are the base for the cloud archive.
Think of TargetCollection as a directory, where your images are stored. A TargetCollection forms a logical group, which is searched as a whole. Of course you can have several TargetCollections in your account, each consisting up to 50,000 images each.
What is the difference between Cloud Archive and Target Collection
Structure of a TargetCollection
Property | Type | Description |
---|---|---|
id | (String) | An ID that uniquely identifies the TargetCollection |
name | (String) | The Name of the TargetCollection, as defined by the user |
creDat | (Number) | A timestamp when the TargetCollection was created (as returned by JavaScript's Date.now() function) |
modDat | (Number) | A timestamp when the TargetCollection was last modified (as returned by JavaScript's Date.now() function) |
Create a Target Collection
Creating a Target Collection is easy and can be done without any prerequisites in your account. In general it is your starting point and most likely your very first action.
Call the endpoint (using the domain of one of the regional servers)
/cloudrecognition/targetCollection
with the mandatory name
field as a POST
request and you will create a new TargetCollection. The response will contain a TargetCollection object, where the ID is most important parameter. You can also add metadata
to a TargetCollection in case you want to some additional descriptive information. The next step is to add images to your TargetCollection, so they can be recognized.
Add Target Images
A Target is an plain image that can be recognized by the Wikitude Cloud Recognition service. Adding or creating a target means to provide a URL to your image to the server, which then downloads the image, analyzes it and adds it to the TargetCollection. To add an image call the endpoint
/cloudrecognition/targetCollection/:tcId/target
with the ID
of the TargetCollection, where you want to add the image. You need to add the a field imageUrl
to your request. The image must be publicly accessible.
Pay attention to the optional fields name
and metadata
. name
is a unique identifier for your target within the TargetCollection. It is up to you to set and use this. The same is true for the metadata
object, which takes a full JSON object and can be filled with any value you like. The metadata
object will be present in the recognition response.
Structure of a Target
Property | Type | Description |
---|---|---|
id | (String) | An ID that uniquely identifies the Target |
name | (String) | The Name of the Target, as defined by the user |
imageUrl | (String) | The URL pointing to the original, uncompressed and uncropped Target binary file |
thumbnailUrl | (String) | The URL pointing to a thumbnail representation of the Target |
rating | (Number) | The rating (from 0 to 3) of the Target |
fileSize | (Number) | The file size of the original Target binary image file, in bytes |
physicalHeight | (Number) | The physical (real world) height of the target, in millimeters |
creDat | (Number) | A timestamp when the Target was created (as returned by JavaScript's Date.now() function) |
modDat | (Number) | A timestamp when the Target was last modified (as returned by JavaScript's Date.now() function) |
metadata | (JSON) | Arbitrary JSON data that is stored together with the target. |
Generate a Cloud Archive
Once you are done with adding targets you need to tell the server that it should generate your TargetCollection into a Cloud Archive. Call
/cloudrecognition/targetCollection/:tcId/generation
again with the ID
of your TargetCollection and the process will be started. Since this call is asynchronous you will receive the response immediately with a path in the Location-property in the header of the response. By calling the url with the path, for example
/cloudrecognition/targetCollection/:tcId/generation/wtc/:generationId
with a GET-method request, you will see the status of the progress of the cloud archive generation in the response body as a JSON object. When the generation is completed, the cloud archive is available for recognition. Note that the generation process can take a while when generating a large TargetCollection for the first time. Small additions to existing cloud archives are processed a lot faster.
- your newly added image will not be recognized
- your deleted image will still be recognized
Your Cloud Archive is now ready on the server and can be used in combination with the Wikitude SDK from your app. See the SDK sample called Cloud Recognition for more details.
Generate a WTC file via API
You can create and download a wtc file of a Target Collection (Generate WTC) with up to 1000 targets by calling
/cloudrecognition/targetCollection/:tcId/generation/wtc
with method POST. You have to specify the SDK version the wtc file should be built for in the request body. Valid values for the version are 3.x
, 4.0
, 4.1
, 5.0
,5.1
,5.2
,5.3
. Optionally, an email address can be added. The email is used for a notification once the generation of the wtc file has finished. Example for the request body:
{
"sdkVersion": "5.0",
"iwantmywtcfile@wikitude-user.com"
}
Similar to the cloud archive generation this call is asynchronous, so the response header (Location
) contains a path useful for requesting the status of the wtc creation. Once the status is COMPLETED
the link to the actual wtc file can be requested from the TargetCollection. The received TargetCollection
object (in the body of the response) contains an additional property called wtc
, which is an array of wtc objects. Those objects consists of the following properties:
- the
url
to the wtc file, - the number of targets (
nrOfTargets
), - the
version
, - the creation date (
creDat
)
Additional calls
Beside the above described steps the Manager API also offers to Delete TargetCollections and Delete Targets.
Using GET
request you can query details about a single TargetCollection, all Target Collections in your account, a single Target and all Targets within a TargetCollection.
The physical height and the metadata of an existing target can be updated.
Migrate
Migration notes for the Wikitude SDK Native API (iOS)
Migrate from 1.4 to 2.0
- New protocol
WTImageTrackerDelegate
New protocol
WTInstantTrackerDelegate
New class
WTAuthorizationRequestManager
- New class
WTTargetCollectionResource
- New class
WTCloudRecognitionService
- New class
WTImageTrackerConfiguration
- New class
WTImageTracker
- New class
WTImageTarget
- New class
WTInstantTracker
- New class
WTInitializationPose
- New class
WTInstantTarget
New class
WTNativeSDKStartupConfiguration
Deprecated class
WTBaseTracker
- Deprecates class
WTClientTracker
. UseWTTargetCollectionResource
+WTImageTracker
instead. Deprecated class
WTCloudTracker
. UseWTCloudRecognitionService
+WTImageTracker
instead.-
- New method
+SDKBuildInformation
- New method
+resolveCaptureDeviceSettings:toSize:targetFrameRate:
- New method
-
- New method
-createTargetCollectionResourceFromURL:completion
- New method
-createCloudRecognitionService:completion
- New method
-createImageTrackerFromTargetCollectionResource:delegate:configuration:
- New method
-createImageTrackerFromCloudRecognitionService:delegate:configuration:
- New method
-createInstantTracker:delegate:configuration:
- Deprecated method
-create2DClientTrackerFromURL:extededTargets:andDelegate:
- Deprecated method
-create3DClientTrackerFromURL:andDelegate:
- Deprecated method
-createCloudTrackerWithToken:targetCollectionId:extededTargets:andDelegate
- New method
WTCaptureDeviceManagerDelegate
- New property
hasTorch
- New property
cameraRenderingTargetTexture
- New property
Migrate from 1.3 to 1.4
iOS 10 now requires a camera usage description. Please make sure that you add the NSCameraUsageDescription
key to your applications .plist.
WTWikitudeNativeSDK
- new
-clearCache
. Use this method to clear all internal SDK caches.
- new
Migrate from 1.0 to 1.1
-
- new
-create2DClientTrackerFromURL:extendedTargets:andDelegate:
Use this method to create a newWTClientTracker
object which is configured for 2D tracking. new
-create3DClientTrackerFromURL:andDelegate:
Use this method to create a newWTClientTracker
object that is configured for 3D tracking.deprecated
-createClientTrackerFromURL:extendedTargets:andDelegate:
Use-create2DClientTrackerFromURL:extendedTargets:andDelegate:
instead.
- new
new
WTTrackingMapRecorder
Use this class to record and save so called 'Tracking Maps' that are needed for 3D tracking.
Reference
iOS Native API Reference
Go to iOS Reference for a complete reference of all iOS Wikitude Native API objects and functions.
Cloud Recognition Manager API
Go to REST API Reference for a complete reference of all REST API calls for the Manager API.
Wikitude SDK iOS Native API Release Notes
Wikitude SDK Native API 2.0.1
Release Date: 15.02.2017
Fixed
- Fixed a memory issue when instant tracking is used in combination with HD frame rendering (not recommended)
Wikitude SDK Native API 2.0.0
Release Date: 25.01.2017
New
- The new instant tracking feature allows users to place augmentations without any markers in any surrounding. Instant tracking uses Wikitudes own SLAM-based 3D engine.
- Added advanced camera settings like HD and 60 fps camera rendering and a manual focus control
- Unified computer vision related class names
Improved
- Updated target collection format (.wtc) which improves the robustness and performance of 2D image recognition.
- Extended tracking improved significant using the new SLAM technology
- Behaviour of removing plugins at runtime - lifecycle callbacks are now called properly
- Plugins now receive a call to
surfaceChanged
in case the device was rotated while the application was not active.
Wikitude SDK Native API 1.4.1
Release Date: 24.11.2016
Fixed
- Fixes an issue where cloud recognition did not work when using input plugins
- Fixes a rare crash in the iOS Native SDK when 3D tracking is used
- Fixes an issue where the Wikitude SDK camera stopped rendering in case an external AVCaptureSession instance was stopped
Wikitude SDK Native API 1.4.0
Release Date: 13.09.2016
New
- Support for iOS 10
- New InputFrameRenderSettings options for Plugins API
Fixed
SDK
- Fixes issues with NAT64 networks
- Fixes a rare crash in the 3D Tracking engine
Improved
- Improved and aligned behaviour of cache handling
- Improved and aligned behaviour of network handling for cloud recognition
Wikitude SDK Native API 1.3.0
Release Date: 08.07.2016
New
- InputPlugins allow developer to use the Wikitude SDK with an external camera feed
Fixed
- Fixes an issue where destroying the SDK while a large .wtc file is loading, crashed the SDK
- Fixes several issues that might lead to a crash if the SDK is paused/resumed
- Fixes several issues that might lead to a crash if the SDK is destroyed and immediately afterwards created again
- Fixes an issue where green camera frames could be rendered under certain circumstances
- Fixes several issues where destroying the SDK might not release all of its allocated memory
Wikitude SDK Native API 1.2.1
Release Date: 22.03.2016 (internal release only)
Fixed
- Unifies camera related API between Android and iOS
Wikitude SDK Native API 1.2.0
Release Date: 01.12.2015
New
- Support multiple regional co-located cloud recognition services
- Improvements in visualization of 3D Tracking Beta
Wikitude SDK Native API 1.1.0
Release Date: 15.10.2015
New
- 3D Tracking Beta for small-sized objects
- Map recorder in sample App
- Updated Client Tracking sample with new 3D Tracking sample
Wikitude SDK Native API 1.0.0
Release Date: 28.08.2015
Wikitude SDK Native API 1.0.0 beta
Release Date: 30.07.2015
New
- Initial Public Release of iOS Native API