Getting started
Welcome to the Wikitude SDK. This document is designed to help you from your very first steps with the Wikitude SDK all the way through to advanced concepts and examples for developing your augmented reality project.
Recommended Usage of this Documentation
The documentation is arranged in a way to guide you through the various steps in your development process. We recommend following each of the steps outlined below and reading the documentation in the order displayed.
Setup your project
In this section we describe the necessary steps to setup a project in a detailed guide.
View the samples
All of the included samples are complete iOS projects and apps. Browse through this section and get an idea of what the SDK is capable of. The relevant parts of the samples are described in more detail to highlight the applied concepts and patterns. These examples are designed to help you get off to a great start with the Wikitude SDK.
Viewing samples based on vision based augmented reality requires the corresponding reference images. All of them are available directly in the description of the sample or available as a collection on this page, which you can either view on the screen or print.
The Wikitude SDK - Augmented Reality for your own app
The Wikitude SDK is a software library and framework for mobile apps used to create augmented reality experiences.
As opposed to the Wikitude SDK - JavaScript API where you define your AR experiences with JavaScript, you will create your AR experiences with ObjC and OpenGL when using the Wikitude Native SDK.
Architecture of the Wikitude SDK
The image above shows the different components of the Wikitude SDK and possible approaches for creating augmented reality apps. Each of these approaches is based on a certain development environments (IDE) and platforms:
- Computer Vision Engine: The computer vision engine is a core component of the Wikitude SDK and used by all platforms. It is not directly accessible, but wrapped either by the Native API or the JavaScript API.
- Wikitude SDK - Native API: Provides access to the Wikitude computer vision engine natively for Android (Java) and iOS (ObjC). It also can load plugins via the Wikitude Plugins API. (NOTE: Wikitude SDK plugins have nothing to do with the Cordova or Unity Plugin concept.)
- Wikitude SDK - JavaScript API: Allows to build augmented reality worlds on basis of HTML and JavaScript. It is available for Android and iOS. The JavaScript API provides access to the functionality of the computer vision engine, location based AR, the Plugins API and dedicated rendering functionality.
- Wikitude SDK - Plugins API: An API to connect your own plugins to the Wikitude SDK.
- Wikitude SDK - Cordova Plugin: On top of the JavaScript API the Cordova plugin allows to use the Wikitude SDK in combination with Apache Cordova.
- Wikitude SDK - Titanium Module: On top of the JavaScript API the Titanium module allows to use the Wikitude SDK in combination with Titanium.
- Wikitude SDK - Unity Plugin: On top of the Native API the Unity plugin allows to use the Wikitude SDK in combination with Unity.
- Wikitude SDK - Xamarin Component: On top of the JavaScript API the Cordova plugin allows to use the Wikitude SDK in combination with Xamarin.
The Wikitude Developer Portal
The Wikitude Developer Section should be your first stop when you have specific development related questions. The portal hosts a very active Developer Community Forum where Wikitude staff members are constantly assisting other developers with helpful tips and advice. A Knowledge Base helps with various questions.
Feedback and Contact
We are always interested in your feedback and suggestions how we can improve this documentation. Please use the contact form on our website or visit us on Google+ or Facebook
Setup Guide iOS Native API
There are only a few steps necessary to add the Wikitude Native SDK to your existing iOS application. This guide will explain them in detail. In general the steps are
- Adding the Wikitude Native SDK .framework to a Xcode project
- Using the main classes to create your augmented reality experiences
Project Setup
Load your Xcode project
The first step is to open an exiting Xcode project. If there is no project already created, do so using the Xcode project configurator.
Adding the Wikitude SDK Framework
To keep things clear, you should copy the Wikitude Native SDK .framework into your Xcode project structure. Having it somewhere else on your machine might lead to confusion when updating to a newer version of the Wikitude Native SDK.
After the .framework was copied, it can be added as 'Embedded Binaries'.
Because the Wikitude Native SDK will fully support Swift, a build setting has to be changed in order to package Swift Standard Libraries into the final application package.
The project is now fully configured to use the Wikitude Native SDK.
Using the Wikitude Native SDK in your Application
After the setup steps are completed, the Wikitude Native SDK is ready to be used within an iOS application.
The Wikitude Native SDK comes with an umbrella header that has to be included in your corresponding source files. The file is called WikitudeNativeSDK.h
and can be imported by calling #import <WikitudeNativeSDK/WikitudeNativeSDK.h>
The main class to work with is WTWikitudeNativeSDK
. It's the central point for creating client or cloud tracker, to obtain an already configured rendering component or to control camera related settings.
The Wikitude Native SDK can be used in trial mode without any license, but to remove the watermark, a valid license has to be set using the -setLicenseKey
method.
Application Lifecycle
To integrate the Wikitude Native SDK into the application lifecycle, use the -start:completion
and -stop
methods of WTWikitudeNativeSDK
. A good place to call them is e.g. a UIViewControllers
-viewWillAppear:
and -viewWillDisappear:
.
To refine the startup phase, the WTStartupConfiguration
object can be used which is the first arguments in first block parameter of -start:completion:
. The second block parameter can be used to check if the Wikitude Native SDK could actually start with the given configuration on the current device. Use the isRunning
parameter to determine the run state and e.g. create tracker in this block. The following listening demonstrates this:
[self.wikitudeSDK start:nil completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
NSURL *clientTrackerURL = [[NSBundle mainBundle] URLForResource:@"magazine" withExtension:@"wtc" subdirectory:@"Assets"];
self.clientTracker = [self.wikitudeSDK.trackerManager createClientTrackerFromURL:clientTrackerURL extendedTargets:nil andDelegate:self];
}
}];
Error Handling
The Wikitude Native SDK calls the WTWikitudeNativeSDK
delegate method -wikitudeNativeSDK: didEncounterInternalError:
as soon as an internally inconsistent state was detected and parts of the SDK might stop working as expected. This is especially helpful while developing the application and not so much for end user notifications.
Client/Cloud Tracker, Camera and Rendering
How client/cloud tracker, the camera controls and rendering can be used, will be explained in the example part of this documentation. Each feature will be explained in detail through a real usage example.
Supported Android Devices
Wikitude SDK (Native API) is running on devices fulfilling the following requirements:
Sensor-based AR (Geo-AR) | Image recognition and tracking |
---|
Requirements for other operating systems and platforms are listed in this overview.
Supported Devices
Wikitude SDK is running on devices fulfilling the following requirements:
Sensor-based AR (Geo-AR) | Image recognition and tracking | |
---|---|---|
Android |
|
|
Android (Native API) |
|
|
Epson |
|
|
Google Glass |
|
|
iOS (JavaScript API) |
|
|
iOS (Native API) |
|
|
Vuzix |
|
|
How to obtain a free trial license
The Wikitude SDK requires a valid license key to be able to run properly. An empty or missing license key will block the augmented reality view from showing any meaningful content. You will see a watermark across the screen with the words License Key Missing
. All JavaScript API calls will be ignored and not interpreted.
When downloading the Wikitude SDK you will be forwarded to the license generation page, where a trial license key is automatically generated for you.
Copy the key into your app, which will unlock the trial mode of the Wikitude SDK. The trial mode of the Wikitude SDK contains the full feature set of the Wikitude SDK but will show a Trial
watermark across the screen.
Each trial license key is valid for every application ID on every operating system. You can use the same trial license key in multiple apps.
Where should I enter the license key
iOS Native API
To use the Wikitude iOS SDK with a certain license key, use the method -setLicenseKey:
defined in WTWikitudeNativeSDK.h
and provide a valid license key.
Examples
The following examples should give you an overview of the capabilities offered by the Wikitude SDK. Each sample is capable of running without modifications on all supported platforms.
The Sample App is a fully functional sample project either for Android or iOS. You can easily import it into Android Studio or Xcode and work from there. The user interface is kept very simple and shows a list to select the sample you are interested.
Examples Application
This section describes the SDKExamples
application in detail and highlights the main features and use-cases of the Wikitude SDK.
Please note that the Xcode projects deployment target is set to iOS 8 and the application has to be deployed to a actual iOS device and not the simulator.
Structure
The Native Examples
application is structured as a table view where each cell launches an individual example, describing a specific feature of the SDK.
The application uses an UINavigationViewController
and pushes each example with a dedicated view controller who implements a complete use case. All view controller are grouped together in the Controller
group. Example specific assets are locates in the Assets
group. Rendering related classes that are used by multiple examples can be found in CoreServices
and Views
.
Usage
To run the application, open the Xcode project and click Run
.
Client Recognition
This example shows how to recognize images in the viewfinder and overlay it with images.
For a better understanding, here are some terms that will be used in the following and other section of this documentation related to vision-based augmented reality.
Target: A 2D target image and its associated extracted data that is used by the tracker to recognize an image.
Target Collection: An archive storing a collection of 2D targets that can be recognized by the tracker. A target collection can hold up to 1000 targets. Target collections are stored as
.wtc
files3D Tracking Map: A tracking map is the equivalent of a target for 3D tracking. The map contains the relevant characteristics of a three-dimensional object. In order to recognize and track 3D objects you need to record a map first and then load this map. Maps are stored as
.wtm
files.ClientTracker: The tracker analyzes the live camera image and detects the 2D targets stored in its associated target collection. Multiple trackers can be created, however only one tracker can be active for recognition at any given time.
Simple 2D Client Tracking iOS
In this section we will go through the WTSimpleClientTrackerViewController code, which you can find in the example applications folder Native Examples/Controller/ClientTracking
. We will discuss general concepts on how to use the Wikitude Native SDK as we go along, please don't skip this sample even if you are only interested in cloud recognition for example.
Please refer to the Setup Guide section of this documentation to find detailed information on how to integrate the Wikitude Native SDK into a Xcode project.
In this example we will use external rendering, for details on how to setup the rendering and the difference between internal and external rendering please read through the section on rendering.
The WTWikitudeNativeSDK
class is structured to be used in a standard iOS view controller as a model of the MVC pattern. This class is central starting point for anything related to the Wikitude Native SDK. It provides factory methods to create different kinds of tracker and to receive information of those through there delegate objects.
A typical usage would be to have a strong reference to a WTWikitudeNativeSDK
object in a view controller and to define this view controller to be conform to the WTWikitudeNativeSDKDelegate
protocol
@interface WTSimpleClientTrackerViewController () <WTWikitudeNativeSDKDelegate, WTClientTrackerDelegate>
The WTWikitudeNativeSDKDelegate
protocol defines methods that can be used to react on SDK specific events. It is the chosen way to communicate e.g. runtime errors that can not be returned directly from an API call. Such errors might contain information about camera authorization status changes and inconsistent rendering setups.
To receive client tracker information about recognized/tracked or lost targets, this view controller is also conform to the WTClientTrackerDelegate
protocol. All methods in the mentioned protocol are optionals thus only provide information from the SDK but don't require any action from the application.
Additionally this protocol provides information about the loading status of a particular client tracker object.
The next step we will take is create an instance of the WTWikitudeNativeSDK
class. This is the only object that you have to create manually. All other object will be created by factory methods of the WTWikitudeNativeSDK object.
WTWikitudeNativeSDK
manually. All object somehow play together in a well defined manner and that why they are internally controlled by the WTWikitudeNativeSDK
object and its sub components like WTTrackerManger
and WTCaptureDeviceManager
.
A WTWikitudeNativeSDK
object can be instantiated at any time and does not depend on any particular application lifecycle event. To integrate internal processes into the hosting application, -start:completion
and -stop
are used. These two methods should either be called from - viewWillAppear: | - viewWillDisappear:
or -viewDidAppear: | -viewDidDisappear:
.
To refine the startup behaviour of the Wikitude Native SDK, the first block parameter of -start:completion:
contains an already created and setup object of type WTStartupConfiguration
. Public properties of this object can be used to further define how the Wikitude Native SDK should start e.g. its camera. Typical settings from this object are the camera preset which defines in which resolution the device camera should be started or what's the desired target rendering frame rate.
The second block parameter contains a BOOL indicating if the given properties could be taken into account during the final startup phase. If an error occurred, isRunning
is set to NO
and the given NSError
object contains more information about what went wrong.
The second block parameter is also the ideal starting point to create any tracker object from the Wikitude Native SDK if the running state is YES.
To load client tracker objects, a target collection is needed. These target collections are represented through a .wtc file (Wikitude Target Collection). The .wtc file can be packed into the application bundle and a file URL retrieved using the NSBundle
method -URLForResource:withExtension:subdirectory:
or uploaded to a server and downloaded from the Wikitude Native SDK.
To get notified when the client tracker finished loading, its delegate method -clientTracker:didFinishedLoadingTargetCollectionFromURL:
is called or -clientTracker:didFailToLoadTargetCollectionFromURL:withError:
if the .wtc could not be loaded.
[self.wikitudeSDK start:^(WTStartupConfiguration *startupConfiguration) {
startupConfiguration.captureDevicePosition = AVCaptureDevicePositionFront;
} completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
NSURL *clientTrackerURL = [[NSBundle mainBundle] URLForResource:@"magazine" withExtension:@"wtc" subdirectory:@"Assets"];
self.clientTracker = [self.wikitudeSDK.trackerManager createClientTrackerFromURL:clientTrackerURL extendedTargets:nil andDelegate:self];
}
}];
Next we will have a closer look on how to get on tracker specific data. The WTClientTrackerDelegate
provides three methods that will be called when the tracker found, lost or tracked targets.
-baseTracker:didRecognizedTarget:
is called if a new target was found in the current camera frame-baseTracker:didTrackedTarget:
is called when an already known target moved to a new position-baseTracker:didLostTarget:
is called when an already known target could not be found anymore in the latest camera frame
The WTImageTarget object provides information about which target changed and what changed for the target, e.g. its position which would be represented through a OpenGL like model view matrix.
- (void)baseTracker:(nonnull WTBaseTracker *)baseTracker didRecognizedTarget:(nonnull WTImageTarget *)recognizedTarget
{
NSLog(@"recognized target '%@'", [recognizedTarget name]);
_isTracking = YES;
}
- (void)baseTrakcer:(nonnull WTBaseTracker *)baseTracker didTrackTarget:(nonnull WTImageTarget *)trackedTarget
{
[self.renderableRectangle setProjectionMatrix:trackedTarget.projection];
[self.renderableRectangle setModelViewMatrix:trackedTarget.modelView];
}
- (void)baseTracker:(nonnull WTBaseTracker *)baseTracker didLostTarget:(nonnull WTImageTarget *)lostTarget
{
NSLog(@"lost target '%@'", [lostTarget name]);
_isTracking = NO;
}
- (void)clientTracker:(nonnull WTClientTracker *)clientTracker didFinishedLoadingTargetCollectionFromURL:(nonnull NSURL *)URL
{
NSLog(@"Client tracker loaded");
}
- (void)clientTracker:(nonnull WTClientTracker *)clientTracker didFailToLoadTargetCollectionFromURL:(nonnull NSURL *)URL withError:(nonnull NSError *)error
{
NSLog(@"Unable to load client tracker. Reason: %@", [error localizedDescription]);
}
Extended 2D Client Tracking iOS
Based on the above to enable Extended Tracking for a tracker you need to provide a String array which defines which targets should be extended. In the sample we simply set a wildcard *
so that all targets in this tracker are extended.
self.clientTracker = [self.wikitudeSDK.trackerManager createClientTrackerFromURL:clientTrackerURL extendedTargets:@[@"*"] andDelegate:self];
3D Client Tracking iOS
In this example we will take a look at how to use the WTTrackingMapRecorder
and 3D tracking configured WTClientTracker
to do 3D Tracking. If you haven't already done so please read through the first section on 2D Tracking before continuing here. Essential concepts like how to setup rendering or how to use a Wikitude Tracker are explained there and won't be repeated here. Similar if you don't already know why we need to record a Tracking Map or what to expect from 3D Tracking in general please read through the introduction to 3D tracking.
Depending of the use case, an App that uses the Wikitude 3D Tracking technology might include an already recorded Tracking Map or needs to create such a Map each time the App is used. If a certain object should be recognisable, than it is not necessary to let the user record a new Tracking Map each time the App is used. A map can be recorded using this example applications Record Map
functionality and packaged into the final App. As this example does not require a certain object to be available for 3D Tracking, a new Tracking Map is recorded every time, including objects that are currently available in your vicinity.
The first change in this example is already at the end of -viewDidLoad
. The delegate object of WTTrackingMapRecorder
is set to this view controller. Objects of type WTTrackingMapRecorder
should not be created manually but only retrieved through the factory method of a WTWikitudeNativeSDK
object. The WTTrackingMapRecorderDelegate
protocol offers methods to get information about a running Tracking Map recording session. We use those methods in this example to indicate when a new Tracking Map contains enough information and the recording session can be stopped.
- (void)viewDidLoad
{
[super viewDidLoad];
// ... Wikitude SDK initialization ...
[[self.wikitudeSDK trackingMapRecorder] setDelegate:self];
}
To start a Tracking Map recording session, WTTrackingMapRecorder
offers -startTrackingMapRecording:
. This method calls a block of type WTTrackingMapRecordingStartupHandler
once the recording session started. The block object offers more information about if the session could be started and if not, why. The Tracking Map recording is started once the start button of a UIAlertAction is tapped.
- (void)startTrackingMapRecordingInternal
{
[self setNavigationItemPrompt:nil withNavigationBarTintColor:nil];
if ( ![[self.wikitudeSDK trackingMapRecorder] isRecording] )
{
__weak typeof(self) weakSelf = self;
[[self.wikitudeSDK trackingMapRecorder] startTrackingMapRecording:^(BOOL isRecording, NSError * _Nullable error) {
if ( !isRecording )
{
// ... react to start issues. The NSError objects contains more information about the issue
}
else
{
// ... the recording session is running and a Tracking Map is recording ...
}
}];
}
}
During the recording, the WTTrackingMapRecorderDelegate
method -trackingMapRecorder:didChangeTrackingMapRecordingQualityFrom:toQuality:
is called every time the quality of the new Tracking Map changes. Recording should continue until enough information could be gathered. The more informations are included in a Tracking Map, the more likely it is to recognise objects from the recorded scene in a later detection phase. In this example, the navigation bars tint color and prompt are updated accordingly.
- (void)trackingMapRecorder:(WTTrackingMapRecorder *)trackingMapRecorder didChangeTrackingMapRecordingQualityFrom:(WTTrackingMapRecordingQuality)oldTrackingQuality toQuality:(WTTrackingMapRecordingQuality)newTrackingQuality
{
UIColor *trackingMapRecordingQualityBarTintColor = nil;
NSString *trackingMapRecordingQualityText = nil;
_currentTrackingMapRecordingQuality = newTrackingQuality;
if ( WTTrackingMapRecordingQuality_Bad == newTrackingQuality )
{
trackingMapRecordingQualityBarTintColor = [UIColor colorWithRed:1 green:0.204 blue:0.125 alpha:1];
trackingMapRecordingQualityText = @"Bad";
}
else if ( WTTrackingMapRecordingQuality_Average == newTrackingQuality )
{
trackingMapRecordingQualityBarTintColor = [UIColor colorWithRed:1 green:0.851 blue:0 alpha:1];
trackingMapRecordingQualityText = @"Average";
}
else
{
trackingMapRecordingQualityBarTintColor = [UIColor colorWithRed:0.420 green:1 blue:0 alpha:1];
trackingMapRecordingQualityText = @"Good";
}
if ( trackingMapRecordingQualityBarTintColor
&&
trackingMapRecordingQualityText )
{
[self setNavigationItemPrompt:[NSString stringWithFormat:@"Recorded Tracking Map Quality: %@", trackingMapRecordingQualityText] withNavigationBarTintColor:trackingMapRecordingQualityBarTintColor];
}
}
To stop a running Tracking Map recording session, -stopTrackingMapRecording:completion:
can be used. This method requires the name of the Tracking Map file that will be written and a WTTrackingMapRecordingCompletionHandler
completion handler that is called once the save operation finished. The completion handler contains more information about if the file could actually be written and, in case of success, also returns it's final file URL.
[[self.wikitudeSDK trackingMapRecorder] stopTrackingMapRecording:@"3d_tracking_map" completion:^(NSURL * _Nullable trackingMapURL, NSError * _Nullable error) {
[weakSelf setNavigationItemPrompt:nil withNavigationBarTintColor:nil];
if ( nil == trackingMapURL )
{
NSLog(@"Unable to save recorded Tracking Map. Reason: %@", [error localizedDescription]);
}
else
{
[weakSelf.navigationItem setRightBarButtonItem:nil animated:YES];
weakSelf.clientTracker = [[weakSelf.wikitudeSDK trackerManager] create3DClientTrackerFromURL:trackingMapURL andDelegate:weakSelf];
}
}];
The Tracking Map save operation is executed asynchrony in a background thread and might take some time to finish. The completion handler is called once this background activity finished.
In this example we use the completion handler to immediately load the recorded Tracking Map using the WTTrackerManager
factory method -create3DClientTrackerFromURL:andDelegate:
. Once the WTClientTracker
object is created, it can be used as an already known 2d WTClientTracker
.
Cloud Recognition
This example shows how to recognize images on a cloud server and then overlay it with augmentations utilizing the CloudTracker
class.
For a better understanding, here are some terms that will be used in the following and other sections of this documentation related to vision-based augmented reality.
Target: An image and its associated extracted data that is used to recognize an image.
Target Collection: A group of
targets
that are searched together. Think of it as a directory, which contains all your images you want to search. The Wikitude SDK can work with two different sorts ofTarget Collections
- On-device Target Collection: a static
wtc
file containing the extracted data of your images. Can consist of up to 1,000 images. - Cloud Target Collection: A target collection stored on the Wikitude server. See
Cloud Archive
below. Can consist of up to 50,000 images.
- On-device Target Collection: a static
Cloud Archive: An archive stored on the server that is optimized for cloud-based recognition. It is generated from a
Target Collection
and is used in combination withCloudTracker
.CloudTracker: Instead of analysing and computing the live camera feed directly on the device like the
ClientTracker
, theCloudTracker
will send the image(s) taken by the camera to the Wikitude Cloud Recognition server. The server will then do the hard work of trying to match the image with your targets in the specified cloud archive. Beside the benefit of searching in large image database using theCloudTracker
instead ofClientTracker
has also a positive impact on the general performance in most cases. Especially when using a large target collection and on older devices.
For both Cloud Recognition samples below we will use external rendering if you don't know what that means please go through the section on rendering before starting here.
The CloudTracker
is able to run in two modes, we call them on-click and continuous. In On-Click mode a single recognition cycle will be executed, in continuous mode the recognition will be started in a variable interval.
Both view controller are located in Controller/CloudTracking
.
Regional server endpoints
Before we get started please note that you have to choose which regional-distributed Wikitude server the SDK should contact.
The cloud recognition server region can be selected by calling -setCloudRecognitionServerRegion:
which is a method of WTTrackerManager
. As a parameter, pass a value of WTCloudRecognitionServerRegion
to specify the regional server. Every cloud tracker that was created afterwards, will contact the specified server.
On-Click Cloud Tracking
The starting point for on-click recognition is WTOnClickCloudRecognition
. In -viewDidLoad
we create a new instance of WTWikitudeNativeSDK
and initialize it with a valid license key.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
[self.wikitudeSDK setLicenseKey:kWTLicenseKey];
Then in -viewDidAppear:
the Wikitude Native SDK is started using the -start:completion:
method. If the SDK could be started, a object of type WTCloudTracker
is created using the WTTrackerManager
factory method -createCloudTrackerWithToken:targetCollectionId:extendedTargets:andDelegate
. The returned pointer needs to be retained in order to keep it alive, so assign it to a strong property.
[self.wikitudeSDK start:nil completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
self.cloudTracker = [self.wikitudeSDK.trackerManager createCloudTrackerWithToken:@"b277eeadc6183ab57a83b07682b3ceba" targetCollectionId:@"54e4b9fe6134bb74351b2aa3" extendedTargets:nil andDelegate:self];
}
}];
Creating a cloud tracker object will automatically load it in the background. To get notified when the cloud tracker finished loading, implement the WTCloudTrackerDelegate
protocol. It provides two methods that are called once the cloud tracker finished loading or failed to load.
- (void)cloudTrackerFinishedLoading:(WTCloudTracker * __nonnull)cloudTracker
{
NSLog(@"cloud tracker finished loading");
}
- (void)cloudTracker:(WTCloudTracker * __nonnull)cloudTracker failedToLoadWithError:(NSError *)error
{
NSLog(@"Cloud tracker failed to load with error: %@", [error localizedDescription]);
}
To start a single recognition, call the cloud tracker -recognize:errorHandler:
method. This method takes two block objects as parameter. The first on is called if a successful server connection could be established and the cloud tracker received a valid server response. The second one is called if any kind of error occurred during this communication. To call the -recognize:errorHandler:
method, a UIButton
is set up to call a specific method once it was touched.
- (IBAction)sendCloudRecognitionRequest:(id)sender
{
if ( self.cloudTracker )
{
if (self.cloudTracker.isLoaded)
{
[self.cloudTracker recognize:^(WTCloudRecognitionResponse *response) {
if ( response.recognized )
{
NSLog(@"Recognized target '%@' which has a rating of '%ld'", [response.targetInformations objectForKey:WTCloudTrackerResponseKey_TargetName], (long)[[response.targetInformations objectForKey:WTCloudTrackerResponseKey_TargetRating] integerValue]);
NSLog(@"Associated (custom) metadata: %@", response.metadata);
}
else
{
NSLog(@"No target image found in the analysed camera frame.\nKeep Calm \n\t&\nKeep Looking");
}
} errorHandler:^(NSError *error) {
NSLog(@"Cloud recognition recognition failed. Reason: %@", [error localizedDescription]);
}];
}
else
{
NSLog(@"Cloud tracker is not yet loaded. No request send.");
}
}
}
Please note that only a recognize call is made if the cloud tracker finished loading.
In case the server communication was successful, the success handler is called with an object of type WTCloudRecognitionResponse
that represents the server result. This object contains information if a image target was found or not, its name and, if specified, its associated meta data.
In case of an error, a NSError
object is passed to the errorHandler which contains detailed information about why the communication failed.
Since the WTCloudTrackerDelegate
protocol also conforms to WTBaseTrackerDelegate
, all tracking related methods to draw a augmentation around the recognized target are available as well.
Continuous Cloud Tracking
On-Click recognition is useful in some particular cases, but more often than not you probably want to use continuous recognition. For continuous cloud recognition we set an interval in which the CloudTracker
automatically calls the recognize function.
To start continuous recognition, simply call the -startContinuousRecognitionWithInterval:successHandler:interruptionHandler:errorHandler:
cloud tracker method once it finished loading.
if ( self.cloudTracker && self.cloudTracker.isLoaded )
{
[self.cloudTracker startContinuousRecognitionWithInterval:1.5 successHandler:^(WTCloudRecognitionResponse *response) {
NSLog(@"received continuous response...");
if ( response.recognized ) {
dispatch_async(dispatch_get_main_queue(), ^{
[self.cloudTracker stopContinuousRecognition];
[self.continuousRecognitionButton setTitle:@"Start Continuous Recognition" forState:UIControlStateNormal];
});
}
} interruptionHandler:nil errorHandler:^(NSError *error) {
}];
[self.continuousRecognitionButton setTitle:@"Stop Continuous Recognition" forState:UIControlStateNormal];
}
else
{
NSLog(@"Cloud tracker is not ready yet.");
}
As before, this method is called once a UIButton
was touched.
The first parameter defines in which interval new camera frames should be sent to the server for processing. The second parameter is again a success handler which contains information about a recognized target. The third parameter is a block that is invoked if the given interval is to short for the server to respond. The last parameter is again a error handle which is invoked in case of any error the occurred during the communication.
To stop a continuous recognition session, e.g. once a target was found, simply call the -stopContinuousRecognition
method.
Rendering
This example shows and explain how rendering works in combination with the Wikitude SDK Native API. There are two methods of rendering available in the Wikitude Native SDK. We call them internal and external rendering. Internal means the OpenGL view is setup by the Wikitude SDK and the SDK user can define custom rendering, that is executed by the Wikitude SDK. On the other hand external rendering means the SDK user sets up the OpenGL view and integrates the Wikitude SDK into this rendering setup.
External Rendering
Most of the examples in the Wikitude Native SDK example application use external rendering, simply because it might be the preferred usage by developers. The example application comes with a simple OpenGL ES 2 view (ExternalEAGLView
) and an also very simple renderer (ExternalRenderer
).
To activate external rendering you need to pass WTRenderingMode_External
to the WTWikitudeNativeSDK
method -initWithRenderingMode:delegate
.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
During the Wikitude Native SDK startup phase, which is initialized with a call to -start:completion:
, several methods from the WTWikitudeNativeSDKDelegate
protocol are called. They are needed to prepare the Wikitude Native SDK for external rendering. All methods in the WTWikitudeNativeSDKDelegate
protocol are marked as optional, but the Wikitude Native SDK will call its -wikitudeNativeSDK:didEncounterInternalError:
delegate method if a required method is not implemented or returned an invalid value. The required methods are:
- -wikitudeNativeSDK:didCreatedExternalUpdateHandler:
- -wikitudeNativeSDK:didCreatedExternalDrawHandler:
- -eaglViewSizeForExternalRenderingInWikitudeNativeSDK:
- -eaglContextForVideoCameraInWikitudeNativeSDK:
- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreatedExternalUpdateHandler:(WTWikitudeUpdateHandler __nonnull)updateHandler
{
self.wikitudeUpdateHandler = updateHandler;
}
- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreatedExternalDrawHandler:(WTWikitudeDrawHandler __nonnull)drawHandler
{
self.wikitudeDrawHandler = drawHandler;
}
- (CGRect)eaglViewSizeForExternalRenderingInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
return self.eaglView.bounds;
}
- (EAGLContext *)eaglContextForVideoCameraInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
if (!_sharedWikitudeEAGLCameraContext )
{
EAGLContext *rendererContext = [self.renderer internalContext];
self.sharedWikitudeEAGLCameraContext = [[EAGLContext alloc] initWithAPI:[rendererContext API] sharegroup:[rendererContext sharegroup]];
}
return self.sharedWikitudeEAGLCameraContext;
}
The first two methods provide block objects, that need to be called every frame from the external rendering system. There purpose is to update internal SDK logic and draw the camera stream in OpenGL.
The last one is used to get to know the current OpenGL view size in order to draw the camera frame fullscreen and with the correct aspect ratio.
External rendering also requires a shared EAGLContext
object that is used by the Wikitude Native SDK to issue camera related OpenGL ES 2 calls. This way custom rendering and Wikitude rendering do not influence each other and less conflicts can occur.
An example of how those block objects can be used is the following:
if ( self.wikitudeUpdateHandler
&&
self.wikitudeDrawHandler )
{
self.wikitudeUpdateHandler();
self.wikitudeDrawHandler();
}
// ...external rendering code...
Such a snippet is typically found somewhere in the external render loop.
Internal Rendering
In case no OpenGL ES 2 rendering is already setup in the hosting Wikitude Native SDK application, the Wikitude Native SDK brings its own set of OpenGL ES 2 compatible classes to do so. These classes are WTEAGLView
and WTRenderer
. Objects of those classes can be retrieved through WTWikitudeNativeSDK
methods. To start the Wikitude Native SDK with internal rendering, pass WTRenderingMode_Internal
to the -initWithRenderingMode:delegate:
method.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_Internal delegate:self];
self.wikitudeRenderer = [self.wikitudeSDK createRenderer];
self.wikitudeEAGLView = [self.wikitudeSDK createEAGLView];
After all required objects are created, view and renderer have to be connected through a call to setRenderer:
.
[self.wikitudeEAGLView setRenderer:self.wikitudeRenderer];
After those two objects are connected, the view has to be added to the view hierarchy of your application. This can either be done using Storyboards or programmatically. The Wikitude Native SDK example application does it in code.
[self.view addSubview:self.wikitudeEAGLView];
[self.wikitudeEAGLView setTranslatesAutoresizingMaskIntoConstraints:NO];
NSDictionary *views = NSDictionaryOfVariableBindings(_wikitudeEAGLView);
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"|[_wikitudeEAGLView]|" options:0 metrics:nil views:views]];
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"V:|[_wikitudeEAGLView]|" options:0 metrics:nil views:views]];
To do some custom OpenGL ES 2 calls from outside the Wikitude Native SDK, custom update and draw block objects can be passed to the Wikitude Native SDK. The corresponding delegate methods are called during the startup phase.
- (WTCustomUpdateHandler)wikitudeNativeSDKNeedsExternalUpdateHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
/* Intentionally returning a nil handler here */
return ^(){};
}
- (WTCustomDrawHandler)wikitudeNativeSDKNeedsExternalDrawHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
return ^(){
if ( _isTracking ) {
[self.renderableRectangle drawInContext:[EAGLContext currentContext]];
}
};
}
In this snippet only custom rendering is done and no update logic.
Plugins API
This guide consists of multiple sections, first we discuss Wikitude SDK Plugins in general, than we talk about platform specifics and how to register a plugin with the Wikitude SDK and then we go through each of the sample plugins included with the Wikitude Example Applications.
- About Wikitude SDK Plugins
- Platform Specifics
- Registering Plugins
- QR & Barcode Plugin
- Face Detection Plugin
About Wikitude SDK Plugins
Technically a plugin is a class, either written in C++ or Java, that is derived from the Wikitude Plugin base class. Beside lifecycle handling and options to enable and disable the plugin, the Plugin class has four main methods that you can override cameraFrameAvailable
which is called each time the camera has a new frame, update
which is called after each image recognition cycle as well as startRender
and endRender
which are called before and after the Wikitude SDK does its rendering.
The most important thing to remember when working with plugins is that they need to have a unique identifier! If the attempt is made to register a plugin with an identifier that is already known to the Wikitude SDK, the register method call will return false.
Plugin Base Class
class Plugin {
public:
Plugin(std::string identifier_);
~Plugin();
string getIdentifier() const; // returns a unique plugin identifier
bool processesColorCameraFrames(); // returns true if the plugins wants to process color frames instead of bw
void setEnabled(bool enabled_);
bool isEnabled();
string callJavaScript(string javaScriptSnippet); // evaluates the given JavaScript snippet in the currently loaded ARchitect World context.
protected:
void initialize(); // called when the plugin is initially added to the Wikitude SDK
void pause(); // called when the Wikitude SDK is paused e.g. the application state changes from active to background
void resume(uint pausedTime_); // called when the Wikitude SDK resumes e.g. from background to active state. pausedTime represents the time in milliseconds that the plugin was not updated.
void destroy(); // called when the plugin is removed from the Wikitude SDK
void cameraFrameAvailable(const Frame&; cameraFrame_); // called each time the camera has a new frame
void update(const vector<RecognizedTarget> recognizedTargets_); // called each time the Wikitude SDK renders a new frame
void startRender(); // called before any Wikitude SDK internal rendering is done
void endRender(); // called right after any Wikitude SDK internal rendering is done
protected:
string _identifier;
bool _enabled;
};
With those methods in place your plugin will be able to read the full camera image for your own purpose, where the YUV image is also processed in wikitude’s computer vision engine.
Information about Recognized Targets
In case you have the wikitude SDK running with ongoing image recognition, the plugin API will populate the RecognizedTarget
in the update
method once an image has been recognized. The plugin can then work with class RecognizedTarget
, which wraps the details of the target image in the camera view. With that you can read out the pose of the target image and use it for your purposes. Additionally the call contains the calculated distance to the recognized target
class RecognizedTarget {
public:
const string& getIdentifier() const; // the identifier of the target. The identifier is defined when the target is added to a target collection
const Mat4& getModelViewMatrix() const; // the model view matrix that defines the transformation of the target in the camera frame (translation, rotation, scale)
const Mat4& getProjectionMatrix() const;
const float getDistanceToCamera() const; // represents the distance from the target to the camera in millimeter
};
Passing values from within the plugin to the JavaScript part of your augmented reality experience is done via the addToJavaScriptQueue()
method of the Plugin class. Using this function will execute any JavaScript code in the context of your augmented reality experience.
Platform Specifics
Thanks to Objective-C++, C++ plugins can be loaded very easily on iOS. Simply create a new C++ file, include and derive from wikitude::sdk::Plugin
and create a std::shared_ptr
from your plugin class. The created shared pointer can then be supplied to the Wikitude SDK specific plugin registration/removal API.
To mark a Objective-C file as Objective-C++, change its file extension from .m to .mm or change the type manually to Objective-C++ using Xcodes Identity and Type inspector.
Registering Plugins
The Wikitude Native SDK for iOS offers three methods to register/remove C++ plugins. All of them are accessible through the WTWikitudeNativeSDK
class.
Registering a C++ Plugin
To register a C++ plugin, call the -registerPlugin:
method and pass a std::shared_ptr
wrapping your C++ plugin pointer. The following snippet shows how to define a property for a shared pointer, its initialization and the registration call.
@property (nonatomic, assign) std::shared_ptr<BarcodePlugin> barcodePlugin;
// ...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self); // arguments are passed to the C++ class constructor
// ...
[self.wikitudeSDK registerPlugin:_barcodePlugin];
Removing a C++ Plugin
To remove a already known C++ plugin, call either the -removePlugin:
or -removeNamedPlugin:
method. The first one takes a shared pointer as argument and tries to find a plugin that matches this plugins identifier. The second one tries to remove a C++ plugin based on a string parameter. The second one is interesting because it allows you to not have a property that holds the shared pointer, but to directly call std::make_shared
when calling -registerPlugin:
. If this is done, the hosting application has no reference to the plugin pointer anymore, but can still remove it using its unique identifier (which has to be known by the hosting application/developer).
[self.architectView removePlugin:_faceDetectionPlugin];
//...
[self.architectView removeNamedPlugin:@"com.wikitude.plugin.face_detection"];
Barcode and QR code reader
This samples shows a full implementation of the popular barcode library ZBar into the Wikitude SDK. As ZBar is licensed under LGPL2.1 this sample can also be used for other projects.
ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code.
The C++ barcode plugin is created when WTBarcodePluginViewController
loads its view object.
- (void)viewDidLoad
{
[super viewDidLoad];
//...
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
//...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self);
}
When WTBarcodePluginViewController
starts the Wikitude Native SDK using the -start:completion:
method, the C++ barcode plugin is registered using the -registerPlugin:
method in the completion block.
- (void)viewDidAppear:(BOOL)animated
{
[super viewDidAppear:animated];
//...
[self.wikitudeSDK start:^(WTStartupConfiguration *startupConfiguration) {
//...
} completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
//...
}
else
{
//...
[self.wikitudeSDK registerPlugin:_barcodePlugin];
}
}];
}
Please note that the plugin is only registered when the Wikitude Native SDK could actually start.
Now let's move to the plugins C++ code. First we'll have a look at the BarcodePlugin.h
file. To create the bar code plugin we derive our BarcodePlugin
class from wikitude::sdk::Plugin
and override initialize
, destroy
, cameraFrameAvailable
and update
. We also declare the following member variables: _worldNeedsUpdate
, _image
, _imageScanner
and _presentingViewController
. The _worldNeedsUpdate
variable will later be used as an indicator if we need to display the recognized barcode, _image
and _imageScanner
are classes from zBar
which we'll use to scan for bar codes and _presentingViewController
will hold a pointer to the WTBarcodePluginViewController
from which we call a method every time we recognize a new barcode or QR code.
#include <zbar.h>
#import <WikitudeNativeSDK/Plugin.h>
#import <WikitudeNativeSDK/Frame.h>
#import <WikitudeNativeSDK/RecognizedTarget.h>
@class WTBarcodePluginViewController;
class BarcodePlugin : public wikitude::sdk::Plugin {
public:
BarcodePlugin(int cameraFrameWidth, int cameraFrameHeight, WTBarcodePluginViewController *presentingViewController);
virtual ~BarcodePlugin();
virtual void initialize();
virtual void destroy();
virtual void cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_);
virtual void update(const std::list<wikitude::sdk::RecognizedTarget>& recognizedTargets_);
protected:
int _worldNeedsUpdate;
zbar::Image _image;
zbar::ImageScanner _imageScanner;
WTBarcodePluginViewController *_presentingViewController;
};
In the constructor we set _worldNeedsUpdate
to zero indicating that there is no update necessary and initialize the zBar::Image
member variable passing its constructor the width and height of the camera frame, the image format of Y800
, set its data pointer to null and the data length to zero.
In the initialize
method we configure the zbar::ImageScanner
by calling setConfig
, enabling all supported bar codes. If you are only interested in one or some particular types of codes, first disabling all bar code types and manually enabling each particular type would be the better idea. That way performance could be greatly improved.
void BarcodePlugin::initialize() {
_imageScanner.set_config(zbar::ZBAR_NONE, zbar::ZBAR_CFG_ENABLE, 1);
}
We react to the destroy
event by setting the current data pointer of the zbar::Image
member to null and length to zero.
void BarcodePlugin::destroy() {
_image.set_data(nullptr, 0);
}
The last but most interesting methods are cameraFrameAvailable
and update
. In the cameraFrameAvailable
method we set the data of our previously initialized zbar::Image
member variable to the frame data we just received and the length of the data to frame width * frame height by calling set_data
. We then start the scanning process by calling the scan
method of our zBar::ImageScanner
passing the zBar::Image
member instance. The zBar::ImageScanner::scan
method returns the number of detected bar codes in the image frame, we save this number in a local variable n
. If n
is not equal to the result of the last frame, which we saved to _worldNeedsUpdate
member variable, we know there was a new bar code detected (meaning there was no bar code in the last frame) or that there was a bar code in the last frame and now there isn't. When that's the case, we do another check if there really was a bar code detected this frame and if there was, we call the presentBarcodeResult
method, passing the code content.
void BarcodePlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
int frameWidth = cameraFrame_.getSize().width;
int frameHeight = cameraFrame_.getSize().height;
_image.set_data(cameraFrame_.getLuminanceData(), frameWidth * frameHeight);
int n = _imageScanner.scan(_image);
if ( n != _worldNeedsUpdate ) {
if ( n ) {
zbar::Image::SymbolIterator symbol = _image.symbol_begin();
[_presentingViewController presentBarCodeResult:[NSString stringWithUTF8String:std::string(symbol->get_data()).c_str()]];
}
}
_worldNeedsUpdate = n;
}
Back in the Objective-C class WTBarcodePluginViewController
, the -presentBarCodeResult:
method is implemented like the following:
- (void)presentBarCodeResult:(NSString *)barcodeScanResult
{
if ( barcodeScanResult ) {
dispatch_async(dispatch_get_main_queue(), ^{
[self.barcodeLabel setText:barcodeScanResult];
});
}
}
Please note that the -presentBarcodeResult:
method is not called from the main thread, meaning that we need to schedule any UI related calls on the main thread. This is done by executing a block asynchronously on the dispatch main queue.
To visualize the detected bar code, we use a NSLabel and set its text property to the detected bar code string.
Face Detection
This samples shows how to add face detection to your Wikitude augmented reality experience using OpenCV.
The Face detection plugin example consists of the C++ classes FaceDetectionPlugin
, FaceDetectionPluginConnector
and the Objective-C class WTFaceDetectionPluginViewController
. We will use OpenCV to detect faces in the current camera frame and OpenGL to render a rectangle around detected faces.
FaceDetectionPluginConnector
is used to connect the FaceDetectionPlugin
class with the WTFaceDetectionPluginViewController
. Since this class mainly exists to ease the implementation of a cross platform plugin, we will not go into any implementation details for this class. We also don't go into any OpenCV or OpenGL details. If one is interested in such topics, the source code is part of the Wikitude Native SDK example application.
WTFaceDetectionPluginViewController
handles the face detection plugin creation and registration exactly as described in the previously bar code example. New for this example is a class method that converts UIDeviceOrientations into an int that is used by the face detection plugin to rotate the camera frame in order to always have faces in the right orientation. This class method is then used every time the device changes its orientation.
__weak typeof(self) weakSelf = self;
[[NSNotificationCenter defaultCenter] addObserverForName:UIDeviceOrientationDidChangeNotification object:[UIDevice currentDevice] queue:[NSOperationQueue mainQueue] usingBlock:^(NSNotification *note) {
weakSelf.faceDetectionPlugin->setFlipFlag( [WTFaceDetectionPluginViewController flipFlagForDeviceOrientation:[[UIDevice currentDevice] orientation]] );
}];
Next we have a look at the FaceDetectionPlugin
class. Again we we will leave out implementation details and focus on how we use the plugin itself. In the cameraFrameAvailable
method we use OpenCV to detect faces in the current camera frame which the Wikitude SDK passes to the plugin. We call the observer which is an instance of the FaceDetectionPluginConnector
to notify the view controller about the result. The plugin base class defines startRender
and endRender
, depending on, if you would like to render on top of or below of all rendering the Wikitude SDK does, you choose one of them, or both to override. To render below all Wikitude rendering we choose startRender
and again call the FaceDetectionPluginConnector
instance which in turn calls the view controller. Since we do not react on the result of the Wikitude SDK image recognition we leave update
blank.
... ctor/dtor ...
void FaceDetectionPlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
wikitude::sdk::Size frameSize = cameraFrame_.getSize();
_scaledCameraFrameWidth = cameraFrame_.getScaledWidth();
_scaledCameraFrameHeight = cameraFrame_.getScaledHeight();
std::memcpy(_grayFrame.data,cameraFrame_.getLuminanceData(), frameSize.height*frameSize.width*sizeof(unsigned char));
//... Control Open CV ...
if ( _result.size() ) {
convertFaceRectToModelViewMatrix(croppedImg, _result.at(0));
_observer->faceDetected(_modelViewMatrix);
} else {
_observer->faceLost();
}
}
void FaceDetectionPlugin::startRender() {
_observer->renderDetectedFaceAugmentation();
}
void FaceDetectionPlugin::update(const std::list<wikitude::sdk::RecognizedTarget> &recognizedTargets_) {
/* Intentionally Left Blank */
}
//... other internally used methods ...
To render a frame around detected faces we created an instance of the StrokedRectangle
class which takes care of rendering a rectangle around faces and all targets of the also active ClientTracker
. When the plugin detects, looses or recalculated the projection matrix it will call the appropriate view controller methods which we use to update the StrokedRectangle
instance. If the Plugin decides it is time to render a frame around a detected face it will call -setFaceIsRecognized:atPosition:
. Since the plugin will only call this method in the startRender
method, we know the current thread is the OpenGL thread and are able to dispatch OpenGL calls.
- (void)setFaceIsRecognized:(BOOL)recognized atPosition:(const float*)modelViewMatrix
{
self.faceDetected = recognized;
if (recognized) {
[self.recognizedFaceRectangle setModelViewMatrix:modelViewMatrix];
}
}
- (void)setFaceAugmentationProjectionMatrix:(const float*)projectionMatrix
{
[self.recognizedFaceRectangle setProjectionMatrix:projectionMatrix];
}
Input Plugins API
This guide provides an introduction into the input plugins API of the Wikitude Native SDK and aims to familiarize the reader with its concepts and constraints. Due to the length and complexity of the corresponding example application source code, it will not be presented in its entirety. Relevant and descriptive source code examples will, however, be provided. Since the Input Plugins API is an extension to the Plugins API, we recommend familiarity with it prior to reading this guide.
- About Wikitude SDK Input Plugins
- Input Plugin Base Class
- Concurrency
- OpenGL Context
- Device Orientation
About Wikitude SDK Input Plugins
The input plugins API provides a means to alter the inputs and outputs of the Wikitude Native SDK. For the input case specifically, custom frame data of arbitrary sources can be supplied as an input to the Wikitude SDK Native API for processing. Complementary, for the output case, the default rendering of the Wikitude SDK Native API can be substituted with more advanced implementations. The custom camera example we provide illustrates both principles in a unified use case. A custom camera stream is supplied as an input and a custom rendering effect is used to augment the rendered output.
Input Plugin Base Class
class InputPlugin: public Plugin {
public:
using InputFrameAvailableNotifier = std::function<int(long frameId, std::shared_ptr<unsigned char> frameData)>;
public:
InputPlugin(std::string identifier_);
virtual ~InputPlugin();
virtual FrameColorSpace getInputFrameColorSpace() = 0;
virtual float getHorizontalAngle() = 0;
virtual Size<int> getInputFrameSize() = 0;
virtual bool requestsInputFrameRendering();
virtual bool requestsInputFrameProcessing();
void notifyNewInputFrame(long frameId_, std::shared_ptr<unsigned char> inputFrame_, bool managedFromOutside_ = false);
InputFrameRenderSettings& getRenderSettings();
virtual void prepareRenderingOfInputFrame(long frameId_);
virtual std::shared_ptr<unsigned char> getPresentableInputFrameData();
virtual void internalError(const std::string& errorMessage);
void setInputFrameAvailableNotifier(InputFrameAvailableNotifier newInputFrameAvailableNotifier);
private:
InputFrameAvailableNotifier _newInputFrameAvailableNotifier;
InputFrameRenderSettings _renderSettings;
std::unique_ptr<InputFrameBufferController> _inputFrameBufferController;
};
The keen observer will be quick to notice that the InputPlugin
class is derived from the Plugin
class. This allows an InputPlugin
to be handled in the same manner as a regular plugin; therefore Plugin
instantiation and registration are identical and will not be discussed redundantly in this guide. Please refer to the Plugins API guide for a detailed explanation. Instead, we will focus on and demonstrate the use of the newly introduced functions.
Color Space
wikitude::sdk::FrameColorSpace YUVFrameInputPlugin::getInputFrameColorSpace() {
return wikitude::sdk::FrameColorSpace::YUV_420_NV21;
}
The getInputFrameColorSpace
function needs to be overridden to provide a FrameColorSpace
enumeration value. The Wikitude Native SDK currently accepts RGB frame data corresponding to the FrameColorSpace::RGB
value, as well as YUV data in 4:2:0 NV21 format corresponding to the FrameColorSpace::YUV_420_NV21
value. The former implies a frame data size of frameWidth * frameHeight * 3
bytes, while the latter implies a frame data size of frameWidth * frameHeight * 3 / 2
bytes.
Field of View
float YUVFrameInputPlugin::getHorizontalAngle() {
return [_camera fieldOfView];
}
The getHorizontalAngle
function needs to be overridden to provide a float
value representing the horizontal field of view angle in degrees of the camera the provided frame was captured with. This value is required for the Wikitude computer vision engine to be able to accurately recognise and track targets within the provided frame. Note that the field of view value may significantly differ from device to device, we therefore recommend querying this value from the frame source directly to ensure representative values. For input image files and input video files this value should be discernible from the corresponding meta data; for an input camera stream this value should be accessible though the corresponding camera API.
Frame Size
wikitude::sdk::Size<int> YUVFrameInputPlugin::getInputFrameSize() {
return {[_camera videoDimensions].width, [_camera videoDimensions].height};
}
The getInputFrameSize
function needs to be overridden to provide an object of type wikitude::sdk::Size<int>
containing the input image width and input image height in pixels. Since this value will be constant for many use cases, you may consider hard-coding it to the appropriate values. Alternatively, as with the previous function, we recommend querying the values from either the input file or the input camera device.
Default Frame Rendering
bool YUVFrameInputPlugin::requestsInputFrameRendering() {
return false;
}
The requestsInputFrameRendering
function can be overridden to provide a boolean value indicating whether the input frame data should be processed by the Wikitude SDK Native API or not. The default implementation returns true, meaning that the frame will be rendered using the internal rendering of the Wikitude SDK Native API to present the frame. Should this function be overridden to return false, the responsibility to present the frame becomes that of the InputPlugin
.
Default Frame Processing
bool YUVFrameInputPlugin::requestsInputFrameProcessing() {
return true;
}
The requestsInputFrameProcessing
function can be overridden to provide a boolean value indicating whether the input frame data should be processed by the Wikitude computer vision engine. The default implementation returns true
, meaning that it will be processed. The plugin will be notified of the recognition results through the update function, as is the case for the regular Plugins API. Should this function to be overridden to return false
, the responsibility to perform the desired algorithms becomes that of the InputPlugin
.
Supplying Image Data
void notifyNewInputFrame(long frameId_, std::shared_ptr<unsigned char> inputFrame_, bool managedFromOutside_ = false);
The notifyNewInputFrame
function needs to be called to pass the actual input frame data to the Wikitude SDK Native API. It requires a unique frame identifier of long
type to be supplied, as well as the frame data itself wrapped into a std::shared_ptr<unsigned char>
. It additionally accepts a boolean value indicating whether the default frame caching should be employed or not. The parameter value defaults to false
, meaning the Wikitude SDK Native API default caching will be used. Should you prefer to supply your own frame caching mechanism, set this value to true
. The default caching mechanism keeps up to 5 recent frames in memory to ensure smooth processing performance. Note that you may need to invoke this method from native code due to the file resource or input stream device only being accessible therein. For this use case we recommend having a look at the custom camera sample application code of the Wikitude Native SDK example application.
Rendering Configuration
InputFrameRenderSettings& getRenderSettings();
The getRenderSettings
function behaviour can be altered to provide a parameterised instance of type InputFrameRenderSettings. The default implementation returns the default constructed _renderSettings
member. Should you like to provide render settings to the internal Wikitude Native SDK that differ from the default constructed values, alter the _renderSettings
accordingly before it is returned.
Frame Caching
virtual void prepareRenderingOfInputFrame(long frameId_);
The prepareRenderingOfInputFrame
function is called whenever a frame has been processed to report its identifier. It is, however, only called if requestsInputFrameRendering
has been overridden to return false
and requestsInputFrameProcessing
has been overridden to return true
. The default implementation of this function releases the frame indicated by the received identifier as well as any older frames from the frame cache. In case of a non-default frame caching mechanism, override this method accordingly. An input parameter of -1
identifies the most recent frame.
Receiving Processed Frame Data
virtual std::shared_ptr<unsigned char> getPresentableInputFrameData();
The getPresentableInputFrameData
function can be called to receive the frame data of the most recently processed frame from the default frame cache. Use this method in case requestsInputFrameRendering
has been overridden to return false
in order to retrieve the current frame data to be rendered. When using a custom frame caching mechanism, this function is obsolete.
Error handling
virtual void internalError(const std::string& errorMessage);
The internalError
function gets called whenever an internal error occurs in the Wikitude Native SDK that was not directly related to input plugins. The input parameter provides a description of the error that occurred.
Internal Use Only
void setInputFrameAvailableNotifier(InputFrameAvailableNotifier newInputFrameAvailableNotifier);
The setInputFrameAvailableNotifier
function is called internally and should not be called anywhere else.
Concurrency
When implementing an InputPlugin
, one needs to be aware that its callback functions are invoked concurrently by the Wikitude SDK Native API. It is therefore necessary to protect against race conditions accordingly. We will present two recommended measures to do so: atomic operations and mutual exclusion.
In order to fully utilize the capabilities of the Input Plugin API one must gather data from several asynchronously called member functions, store them, potentially as member variables, and subsequently use them collectively. These operations might be vulnerable to race conditions.
An example snippet from the custom camera example application code:
void YUVFrameInputPlugin::surfaceChanged(wikitude::sdk::Size<int> renderSurfaceSize_, wikitude::sdk::Size<float> cameraSurfaceScaling_, wikitude::sdk::DeviceOrientation deviceOrientation_) {
// some orientation handling code here
_surfaceInitialized.store(true);
}
void YUVFrameInputPlugin::startRender() {
// some early exit code here
render();
}
void YUVFrameInputPlugin::render() {
// some early exit code here
if (!_surfaceInitialized.load()) {
return;
}
// lots of OpenGL code here
}
#include <atomic>
std::atomic_bool _surfaceInitialized;
The surfaceChanged
function and the startRender
function are invoked concurrently. We rely on a boolean value inside the render function that is set from the surfaceChanged
function, yielding a race condition should boolean reads and writes be non-atomic. In such cases, involving intrinsic data types for which atomic operations are provided by the C++ standard library, we recommend their use. These std::atomics
can either be set and read intuitively through the corresponding operators or though the load
and store
functions.
An alternative snippet for which atomic operations are not available:
void YUVFrameInputPlugin::update(const std::list<wikitude::sdk::RecognizedTarget>& recognizedTargets_) {
// platform specific intialization code here
{ // mutex auto release scope
std::lock_guard<std::mutex> lock(_currentlyRecognizedTargetsMutex);
_currentlyRecognizedTargets = std::list<wikitude::sdk::RecognizedTarget>(recognizedTargets_);
}
}
void YUVFrameInputPlugin::startRender() {
// some early exit code here
render();
}
void YUVFrameInputPlugin::render() {
// early returns and lots of OpenGL code here
{ // mutex auto release scope
std::unique_lock<std::mutex> lock(_currentlyRecognizedTargetsMutex);
if (!_currentlyRecognizedTargets.empty()) {
const wikitude::sdk::RecognizedTarget targetToDraw = _currentlyRecognizedTargets.front();
// early unlock to minimize locking duration
lock.unlock();
// lots of OpenGL code here
}
}
}
The update
function and the startRender
function are invoked concurrently. We, again, rely on data being set from an asynchronous function within our render function. Contrary to the previous case though, an object of type std::list
cannot be set atomically using std::atomics
. Therefore we employ a std::mutex
as a locking mechanism to ensure atomicity. As depicted by the code snippet, we encourage the use of RAII style mutex locking using std::lock_guard
and std::unique_lock
to ensure proper mutex release.
OpenGL Context
Another important issue to be aware of is the availability of a valid OpenGL context during plugin run-time. We guarantee such a valid context to be available during the execution of the startRender
, endRender
, pause
and resume
functions. The former two functions should contain all of the rendering related function calls, while the latter should be used to release and acquire OpenGL related resources as the OpenGL context is likely to be destroyed upon pausing the application and recreated upon resuming the application. Therefore all the previously acquired OpenGL handles are no longer valid and need to be reacquired.
A code snipped from the custom camera example:
void YUVFrameInputPlugin::pause() {
releaseFramebufferObject();
releaseFrameTextures();
releaseVertexBuffers();
releaseShaderProgram();
_renderingInitialized.store(false);
// some additional code here
}
void YUVFrameInputPlugin::startRender() {
if (!_renderingInitialized.load()) {
_renderingInitialized.store(setupRendering());
}
render();
}
We release all the OpenGL resources we previously created and atomically set the _renderingInitialized
flag to false
, causing the rendering environment to be reinitialised during the next execution of the render loop.
Device Orientation
Lastly, we will demonstrate the rendering of an input frame from within an InputPlugin
using OpenGL with device orientations taken into account. While there are alternative ways to achieve the correctly oriented frame renderings, we recommend applying the required transformations as matrices within a custom vertex shader.
We propose the following code in our custom camera example application to compose a matrix that is to be applied to a fullscreen quad:
void YUVFrameInputPlugin::surfaceChanged(wikitude::sdk::Size<int> renderSurfaceSize_, wikitude::sdk::Size<float> cameraSurfaceScaling_, wikitude::sdk::DeviceOrientation deviceOrientation_) {
wikitude::sdk::Matrix4 scaleMatrix;
scaleMatrix.scale(cameraSurfaceScaling_.width, cameraSurfaceScaling_.height, 1.0f);
switch (deviceOrientation_)
{
case wikitude::sdk::DeviceOrientation::DeviceOrientationPortrait:
{
wikitude::sdk::Matrix4 rotationToPortrait;
rotationToPortrait.rotateZ(270.0f);
_orientationMatrix = rotationToPortrait;
break;
}
case wikitude::sdk::DeviceOrientation::DeviceOrientationPortraitUpsideDown:
{
wikitude::sdk::Matrix4 rotationToUpsideDown;
rotationToUpsideDown.rotateZ(90.0f);
_orientationMatrix = rotationToUpsideDown;
break;
}
case wikitude::sdk::DeviceOrientation::DeviceOrientationLandscapeLeft:
{
wikitude::sdk::Matrix4 rotationToLandscapeLeft;
rotationToLandscapeLeft.rotateZ(180.0f);
_orientationMatrix = rotationToLandscapeLeft;
break;
}
case wikitude::sdk::DeviceOrientation::DeviceOrientationLandscapeRight:
{
_orientationMatrix.identity();
break;
}
}
_modelMatrix = scaleMatrix * _orientationMatrix;
// some synchronization code here
}
attribute vec3 vPosition;
attribute vec2 vTexCoords;
varying mediump vec2 fTexCoords;
uniform mat4 uModelMatrix;
void main(void)
{
gl_Position = uModelMatrix * vec4(vPosition, 1.0);
fTexCoords = vTexCoords;
}";
struct Vertex
{
GLfloat position[3];
GLfloat texCoord[2];
};
Vertex _vertices[4];
_vertices[0] = (Vertex){{1.0f, -1.0f, 0}, {1.0f, 0.0f}};
_vertices[1] = (Vertex){{1.0f, 1.0f, 0}, {1.0f, 1.0f}};
_vertices[2] = (Vertex){{-1.0f, 1.0f, 0}, {0.0f, 1.0f}};
_vertices[3] = (Vertex){{-1.0f, -1.0f, 0}, {0.0f, 0.0f}};
The matrix composed within the surfaceChanged
function is supplied to the vertex shader as a uniform
parameter and subsequently used to transform the input vertices. Be aware though, that an additional matrix may be required depending on whether you previously rendered to a FBO. If so, the following amendment to the surfaceChanged
function should correct the flipped Y-axis resulting from this process:
wikitude::sdk::Matrix4 scaleMatrix;
_fboCorrectionMatrix.scale(1.0f, -1.0f, 1.0f);
// same device orientation code here as depicted above
_modelMatrix = scaleMatrix * _orientationMatrix * _fboCorrectionMatrix;
For a complete implementation of an input plugin for a specific and advanced use case, we strongly recommend looking into the custom camera example application source code. Additionally, the custom camera sample source code is an excellent starting point to build your own implementation from.
Camera Controls
The WTCaptureDeviceManager
allows you to change capture device specific settings during an active capture session. It lets you change e.g.
* the capture device position
* the capture device focus mode
* the capture device zoom level
A valid pointer to a WTCaptureDeviceManager
object can be retrieved from the WTWikitudeNativeSDK
method -captureDeviceManager
. Please note that this pointer is only valid if the native SDK is running. If the SDK is currently stopped, changes for the capture device are done using the WTStartupConfiguration
object that is passed as argument to the startupHandler when calling -start:completion:
.
The sample uses a UIPickerView
to let the user select between different camera specific values. This UIPickerView
is added to the view hierarchy in the main storyboards camera scene. By default, the picker view is placed behind the OpenGL ES 2 view, so that it is not visible. To show the picker view, the bottom OpenGL ES 2 view constraint is modified and animated, if a button is touched, to unveil the picker view. A standard iOS SDK UIView animation is used to do so. Please refer to the WTCameraControlsViewController
-handleCameraControlsPresentation
method.
The connection between the UIPickerView
and WTCaptureDeviceManager
is done by a custom class called WTCameraControlsModel
and its delegate object (Which is in this case WTCameraControlsViewController
). This custom class WTCameraControlsModel
implements the UIPickerView
data source and delegate protocol and uses its custom delegate object to retrieve current camera specific values or exposes newly selected values from the picker view.
Implementation details of WTCameraControlsModel
are not discussed in this guide as it only handles UIKit related functionality. The following snippets shows the previously mentioned object connection. All following snippets can be found in the WTCameraControlsViewController
class.
self.cameraControlsPickerModel = [[WTCameraControlsModel alloc] init];
self.cameraControlsPickerModel.delegate = self;
self.cameraControlPicker.dataSource = _cameraControlsPickerModel;
self.cameraControlPicker.delegate = _cameraControlsPickerModel;
The camera controls model knows internally about all possible camera settings and uses these information to load the picker view. Once the picker is loaded and the user selects a different value, the camera controls model delegate is used to propagate the value change. Since the camera controls view controller implements this delegate protocol and also holds a reference to the Wikitude Native SDK, which by itself provides access to the capture device manager), the view controller sets the appropriate capture device value when this camera controls model delegate is called. The following snippet demonstrates this for the capture device position:
- (void)cameraControlsModel:(WTCameraControlsModel *)cameraControlsModel didSelectCameraPosition:(AVCaptureDevicePosition)activeCameraPosition
{
if ( [self.wikitudeSDK captureDeviceManager] ) {
[[self.wikitudeSDK captureDeviceManager] setActiveCaptureDevicePosition:activeCameraPosition];
}
}
Please note that there is a nil check when [self.wikitudeSDK captureDeviceManager]
is called. This is because the Wikitude Native SDK could be paused when the picker changes one of its values.
Every time the picker becomes visible again, the camera controls model asks through its delegate object which capture device settings are currently active. The following snippet again demonstrates this for the currently active capture device position.
- (AVCaptureDevicePosition)currentlyActiveCameraPositionForCameraControlsModel:(WTCameraControlsModel *)cameraControlsModel
{
return [[self.wikitudeSDK captureDeviceManager] activeCaptureDevicePosition];
}
Wikitude 2D Tracking
In this section of the documentation we are covering tools shipped with the Wikitude SDK, which help you in your development.
Target Management
This guide gives you an overview of how to create a target collection that you can use to detect and track images within your augmented reality experience.
In general the conversion can be done via four different tools:
- Wikitude Studio: A browser based tool to convert your images to a wtc file. You can find the tool under: https://targetmanager.wikitude.com. You need your free developer account to log-in. This tool is described in more detail further below
- WTC Editor within Unity Editor: The Wikitude Unity plugin installs a WTC Editor as extension of the Unity Editor. Unity developers can manage all their targets and target collections directly within Unity Editor.
- RESTful API. Make use of all features provided by the web tool via direct server endpoints.
- Targets Enterprise Script: A binary shell script available for Mac OS X and Linux converting images to target collections. Pleases contact Wikitude Sales team for technical requirements and pricing.
The following images describes the relationship between the above mentioned methods and the Wikitude Cloud Recognition Service, which is not scope of this documentation.
Web Targetmanager - Wikitude Studio
Add a new project
- Open https://targetmanager.wikitude.com and login with your Wikitude developer account
- Add a new project to your project collection
Add target images
- Enter an existing project
- Add new target images to the project either by clicking on
Add Targets
or drag & drop them on the empty area. Supported file formats include PNG and JPEG. If you are using PNG images, please make sure that it does not contain any transparent pixels, only solid coloured images are supported.
- When uploading a target the file name is used as
target name
. It identifies a target in your experience. If thetarget name
is not completely visible, hover over it to reveal the full name or double click the target to enter edit-mode.
If you add your own target images you need the target name to set them in
AR.Trackable2DObject
.Star Rating
- 0 stars: Not suitable for tracking. This target image cannot be tracked because it lacks textured features with high local contrast. Please consider choosing another target image.
- 1 star: Limited tracking ability. This target image provides basic tracking performance in good lightning conditions. Please consider improving the image
- 2 stars: Good tracking ability. This target image will track well in most conditions.
- 3 stars: Very good tracking ability. This target image will track very well in most conditions.
General advice for reference images
- Good image characteristics:
- Diversely textured image with high local contrast
- Bad image characteristics:
- Large areas with solid color or smooth color transitions
- Repetitive patterns
- Logos, signs
Create a WTC file
- AR.ClientTracker requires a WTC (Wikitude Target Collection) file which contains all information of the targets that should be recognized. Enter the project you need the file for and click the WTC icon in the toolbar.
- Select the Wikitude SDK version you're using and click Generate to trigger the creation of the WTC file. You will be notified via e-Mai once the file is available for download.
Use project's WTC file in your ARchitect World
Look at one of the client recognition examples or refer to the JavaScript API reference of AR.ClientTracker
for instructions on how to use the created target collection for augmentations in your ARchitect Worlds.
Cloud Recognition
Any existing project may also be published to the Cloud to make it accessible for AR.CloudTracker
.
Click the Cloud icon in the toolbar for more details.
Cloud Recognition is available for free in your testing process but you must purchase a license for productive use. Learn more
Once a project is published it is accessible via Wiktiude SDK using 'Client Token' and 'Target Collection ID' (compare AR.CloudTracker
)
Hints
You may unpublish a project at any time but be aware that this action has immediate effect on your application(s) making use of the credentials.
Metadata in the 'Edit Target' dialog is solely relevant for Cloud Recognition whereat Physical Height is only relevant for distanceToTarget feature.
Leave Physical Height empty if you do not use the distanceToTarget feature of
AR.Trackable2DObject
.The Metadata field is very useful. It allows you to attach JSON data to a target. That way you can define any kind of additional data and react on it dynamically in the SDK to e.g. let a button refer to a details page which is defined in the Metadata JSON.
Best practice for target images
This guide gives you an overview of how to create a target collection that you can use to detect and track images within your ARchitect World.
Summary
Preferred images have:
- between 500 to 1000 pixels in each dimension
- Rich contrast
- Evenly distributed textured areas
- Many corner like structures
Unsuitable images have:
- Smaller dimensions than 500 pixels
- Larger than 1000 pixels as they do not provide more accurate results
- Large amounts of text
- Many repetitive patterns
- Large single-colored areas
- Color contrast only e.g. green to red edge), because all images are processed as grayscale images
Optimal Image Dimensions
- Optimal images are sized between 500 and 1000 pixels in each dimension
- Small images do not contain enough graphical information to extract so called feature points. The uniqueness, amount and distribution of features points are the key indicators for good detection and tracking quality
- Larger images do not improve the tracking quality
Low contrast images
- Images with high local contrast and large amount of rich textured areas is best suited for reliable detection and tracking
- Color contrast only (i.e. green to red edge) appears as high contrast to the human eye but is not discriminative to computer vision algorithms as they are operating on grayscale images Tip: For low contrast images, try to increase the contrast of your target image with an image editing tool like Gimp or PhotoShop to improve detection and tracking quality
Distribution of textured areas
- Images with evenly distributed textured areas are good candidates for reliable detection and tracking
- This might be the hardest part to be in control of and often can’t be changed. Tip: Try to crop the most prominent part of your image and use only this as target image.
Images with whitespace
- Single-colored areas or smooth color transitions often found in backgrounds do not exhibit graphical information suitable for detection and tracking. Tip: Try to crop the most prominent part of your image and use only this as target image.
Vector-based graphics
- Logos and vector-based graphics usually consist of very few areas with high local contrast and textured structures and are therefore hard to detect and track.
Tip: Try to add additional elements to the graphic like your logotype or any other specific elements, which can go along with your graphic.
Images with a lot of text
- Images consisting primarily of large areas of text are hard to detect and track.
Tip: Try to have at least some graphical material and images next to your text for your target image.
Repetitive patterns
- Repetitive patterns exhibit the same graphical information information at each feature point and therefore cannot be localized reliably
- Images with slightly irregular structures can convey a similar information to the target audience while providing enough unique feature points to be detected (second image)
Tip: Try a different selection of your image including non pattern parts or use images with irregular patterns
Target Images
All samples
Click here to download all target images
Wikitude 3D Tracking
About Wikitude 3D Tracking
While the Wikitude SDK and its own integrated computer vision engine have been excelling over the past years to detect planar images, our goal was always to not stop at the 2nd dimension but extend recognition and tracking to the third dimension.
The new 3D computer vision engine included in the Wikitude SDK (Native API) can be used to recognize and track arbitrary three-dimensional objects and pieces. The 3D computer vision engine now captures and tracks the depth and distance of salient points in the live camera image. When saved as a tracking map, these points can be recognized and tracked at a later stage again.
We recommend to start to study the sample Client Recognition > 3D Tracking
where we are describing how to use this feature in your augmented reality experience. The sample describes the full flow of how to record a map, save and load a map and use the updated class TrackerManager
. You can read more about how to record 3D tracking maps in a separate guide and understand which objects and scene work well using the Wikitude 3D Tracking engine from our 3D Tracking Guidelines and best practices. For details on the actual classes checkout the reference.
License
For commercial use, Wikitude 3D tracking is part of the SDK PRO family (SDK PRO, SDK PRO+ and SDK PRO+ Unlimited).
Limitations of the Wikitude 3D Tracking Beta
- The functionality is currently only available in the Native API for Android and iOS
- Only small-size tracking maps (scenes) are supported in the SDK and will work well - see the chapter 3D Tracking Guidelines and best practices for more details on well suited objects
- It is not possible to combine 2D and 3D tracking
- A tracking map cannot be edited after it was recorded
Creating 3D Tracking Maps
To be able to use 3D Tracking in your Wikitude powered application, you will need to create a so called Tracking Map. This Tracking Map is a simple file (.wtm file) which contains all the information needed by the Wikitude SDK to track the 3D objects of your choice. Similar to a .wtc file you might know from 2D tracking you later have the choice of packaging the Tracking Map file within your app or store it on a web server and let the Wikitude SDK download it.
Using the Wikitude iOS Example Application to record Tracking Maps
To use the Wikitude SDK (Native API) example application to recored a Tracking Map, download the Wikitude Native SDK release package and open the Xcode project that is located in the Examples/Native\ Examples.xcodeproj.
Note: Tracking Maps are stored in the applications Documents directory.
- Start the Wikitude SDK (Native API) Example Application
- Tap on "Record Map" at the top-right of the screen
- Enter the name for the Tracking Map that you are about to record
- Now position your device so that the objects you would like to track are visible on the screen
- When you are ready click on "Start Tracking Map Recording" to begin the recording
- Move the device around until the quality changes from 'Bad' to 'Average' or 'Good'
- After you captured everything you would like to track with this particular map, click on "Stop Tracking Map Recording"
- Decide if you want to actually save the recorded map or delete it
- If you choose to keep it, connect the device to your Mac, start iTunes and download the .wtm file from within the iTunes File Sharing menu. (Alternatively you can also download the App Container from within Xcode)
Quality of the Tracking Map
The Map recorder and the 3D Tracking sample in the sample app both include a high-level quality indicator, when you record a map. This quality indicator gives you a first estimate how well the tracking performance will be. There are tree levels, which will be displayed
- Bad: There are too few points available to recognize and track the object
- Average: There are few points available to recognize and track the object. This will result in increased shaking and varying recognition results.
- Good: The algorithm has found enough points to recognize and track the object
Check out the guide 3D Tracking Guidelines and best practices how to come to a decent recognition and tracking result.
Editing a recorded Tracking Map
During the beta phase of 3D tracking it is not possible to edit the tracking map after it has been saved.
3D Tracking Guidelines and best practices
The Wikitude SDK 3D Tracking engine currently is optimized for small-sized scenes and objects. With small-sized we mean table-sized scenes or scenes/objects that stretch a few meters. In contrast to other services it has not been optimized to work with particularly tiny objects only measuring a few centimeters
Objects and scenes that 3D Tracking engine can recognize and track well:
- composite scenes with different objects
- highly textured objects
- evenly lit scenes
Objects and scenes where the 3D Tracking engine will not operate well:
- Shiny objects with minimal textures (e.g. solid metal surfaces)
- Tiny objects only measuring a few centimeters
- Often changing illumination and lighting conditions
- Scenes containing of few objects
- White (icy) walls
Wikitude Cloud Recognition
The Wikitude Cloud Recognition service is a cloud-based service provided by Wikitude, which recognizes images sent from Android and iOS apps using the Wikitude SDK. The recognized images are then tracked in the live camera feed and can be used for augmented reality experiences.
This documentation focuses on the RESTful API called Manager API, which is used to interact on a backend level with the Cloud Recognition service.
General Definitions
Target: An image and its associated extracted data that is used to recognize an image.
Target Collection: A group of
targets
that are searched together. Think of it as a directory, which contains all your images you want to search. The Wikitude SDK can work with two different sorts ofTargetCollections
- On-device Target Collection: a static
wtc
file containing the extracted data of your images. Can consist of up to 1,000 images. - Cloud Target Collection: A target collection stored on the Wikitude server. See
Cloud Archive
below.
- On-device Target Collection: a static
- Cloud Archive: An archive stored on the server that is optimized for cloud-based recognition. It is generated from a
TargetCollection
and is used in combination with the Wikitude SDKAR.CloudTracker
.
Manager API: A RESTful web API allowing developers to interact with the Cloud Recognition server for managing Targets
, TargetCollections
and Cloud Archives
. Only you as a developer uses this API. None of your users of your app will interact with this API.
Client API: The Client API is the interface between the Wikitude SDK and the Cloud Recognition Service. The API itself is encapsulated in the Wikitude SDK class AR.CloudTracker
and not directly accessible. Calls on the client API are called Scans
.
Region: Wikitude is providing several hosting locations for its Cloud Recognition services to cut down unwanted network latency. As a developer you need to choose on which Region
you and your customers want to operate.
Getting Started with the Cloud Recognition Service
Regional availability of Wikitude Cloud Recognition Service
As as a developer using Wikitude Cloud Recognition Service you need to choose which server location you want to use for your projects. Wikitude operates several servers running Wikitude Cloud Recognition Service in different locations world-wide.
As the region servers are separated content which is stored on one region servers is only available on this particular server. Content is not synced across regions. You can choose from the following Regions
Americas
China
Europe
The servers for each region have separate dedicated domain names and therefore different configurations.
Region | Target Manager | Manager API | SDK Setting |
---|---|---|---|
Americas | targetmanager‑us.wikitude.com | https://api-us.wikitude.com | Americas |
China | targetmanager‑cn.wikitude.com | https://api-cn.wikitude.com | China |
Europe | targetmanager.wikitude.com | https://api.wikitude.com or https://api-eu.wikitude.com | Europe |
Preparation
First Steps and General Usage
- Get familiar with the Manager API calls in the API Reference.
- Create a first Target Collection using the
Create Target Collection
endpoint and note down the ID of the Target Collection - Create targets using the
Create Target
endpoint for that particular Target Collection - Important:
Generate a Cloud Archive
for your Target Collection - Go to the Wikitude SDK and create an Android or iOS project
- Use the Client API token to authenticate your Android or iOS project
- Use your Target Collection ID to recognize images
For more information on the available endpoints and how to work with the Manager API see the workflow section.
Instead of creating a TargetCollection
, adding one or more Targets
, and generating a Cloud Archive
by calling the REST API, the Wikitude Targetmanager can be used to perform these steps (1-4) in the browser alternatively.
In case you would like to immediately test the API calls we recommend the tool Postman. It helps you to quickly construct the requests and analyze the responses.
Authentication
The Cloud Recognition Service knows two authentication tokens, that you need in order to work with the service
Manager API token You need this token to authenticate yourself against the RESTful Manager API. The Manager API is used to create, add and delete targets and target collections. The token identifies your developer account. Calls to the Manager API do not count towards your quota limits.
Client API token You need this token to authenticate calls from the Wikitude SDK to the Cloud Recognition services. It again authenticates calls as legitimate. The token is bound to your developer account. Calls from the Wikitude SDK to the service with a wrong or missing token can not access your target collections.
Authentication on the Manager API
The Manager API token must be added to each call towards the Wikitude Cloud Recognition Manager API. The token authenticates the user account that is using the API.
Authentication on the Client API
The Client API token must be added to your app project using the Wikitude SDK. This token is needed additionally beside the SDK license key when working with the AR.CloudTracker
class.
Quota and Limits
General Upload Limit
The Wikitude Cloud Recognition will not accept images bigger than 1024kB (1 MB). Trying to upload images exceeding this file size will result in a HTTP status code 400
together with an error message FILE_SIZE_LIMIT_EXCEED
.
Limits for the Wikitude Cloud Recognition Service
There are two main limitations for the Wikitude Cloud Recognition service that you need to be aware of:
Targets Your token has a certain number of targets that you can upload and store on the cloud service under your developer account. The limit is always counted for your entire developer account and not for a single target collection. The service is not counting single uploads, but how many targets are currently stored in target collections under your account.
Scans Scans are in effect calls from the Wikitude SDK via the Client API to the Cloud Recognition servers. All commercial license come with an allowance of 1,000,000 scans per month per developer account. Note: When using Continuous Search mode multiple calls are made to the server.
Maximum Number of Targets in a Target Collection
A target collection can't exceed 50,000 targets.
Free Trial License for Cloud Recognition
Wikitude provides a trial token for each developer account to try out the Cloud Recognition for free. This trial token has set a quota limit that allows developers to try and test the functionality of the service. Limitations for trial accounts
- Targets: 50,000
- Scans: 1,000 per month
To get your trial token for the REST, please visit the License page. The trial token is directly integrated into the Target Manager Frontend.
Commercial Licenses
For production systems, we offer commercial licenses with various quota limits for purchase.
Product | Targets | Scans |
---|---|---|
Cloud Recognition 1000 | 1,000 | 1,000,000 |
Cloud Recognition 10000 | 10,000 | 1,000,000 |
Cloud Recognition 25000 | 25,000 | 1,000,000 |
Cloud Recognition 50000 | 50,000 | 1,000,000 |
Cloud Recognition 100000 (*) | 100,000 | 1,000,000 |
(*) Maximum number of targets per target collection can't exceed 50,000
Your first Target Collections
Target Collections are central to working with Cloud Recognition service. They keep all your target images and are the base for the cloud archive.
Think of TargetCollection as a directory, where your images are stored. A TargetCollection forms a logical group, which is searched as a whole. Of course you can have several TargetCollections in your account, each consisting up to 50,000 images each.
What is the difference between Cloud Archive and Target Collection
Structure of a TargetCollection
Property | Type | Description |
---|---|---|
id | (String) | An ID that uniquely identifies the TargetCollection |
name | (String) | The Name of the TargetCollection, as defined by the user |
creDat | (Number) | A timestamp when the TargetCollection was created (as returned by JavaScript's Date.now() function) |
modDat | (Number) | A timestamp when the TargetCollection was last modified (as returned by JavaScript's Date.now() function) |
Create a Target Collection
Creating a Target Collection is easy and can be done without any prerequisites in your account. In general it is your starting point and most likely your very first action.
Call the endpoint (using the domain of one of the regional servers)
/cloudrecognition/targetCollection
with the mandatory name
field as a POST
request and you will create a new TargetCollection. The response will contain a TargetCollection object, where the ID is most important parameter. You can also add metadata
to a TargetCollection in case you want to some additional descriptive information. The next step is to add images to your TargetCollection, so they can be recognized.
Add Target Images
A Target is an plain image that can be recognized by the Wikitude Cloud Recognition service. Adding or creating a target means to provide a URL to your image to the server, which then downloads the image, analyzes it and adds it to the TargetCollection. To add an image call the endpoint
/cloudrecognition/targetCollection/:tcId/target
with the ID
of the TargetCollection, where you want to add the image. You need to add the a field imageUrl
to your request. The image must be publicly accessible.
Pay attention to the optional fields name
and metadata
. name
is a unique identifier for your target within the TargetCollection. It is up to you to set and use this. The same is true for the metadata
object, which takes a full JSON object and can be filled with any value you like. The metadata
object will be present in the recognition response.
Structure of a Target
Property | Type | Description |
---|---|---|
id | (String) | An ID that uniquely identifies the Target |
name | (String) | The Name of the Target, as defined by the user |
imageUrl | (String) | The URL pointing to the original, uncompressed and uncropped Target binary file |
thumbnailUrl | (String) | The URL pointing to a thumbnail representation of the Target |
rating | (Number) | The rating (from 0 to 3) of the Target |
fileSize | (Number) | The file size of the original Target binary image file, in bytes |
physicalHeight | (Number) | The physical (real world) height of the target, in millimeters |
creDat | (Number) | A timestamp when the Target was created (as returned by JavaScript's Date.now() function) |
modDat | (Number) | A timestamp when the Target was last modified (as returned by JavaScript's Date.now() function) |
metadata | (JSON) | Arbitrary JSON data that is stored together with the target. |
Generate a Cloud Archive
Once you are done with adding targets you need to tell the server that it should generate your TargetCollection into a Cloud Archive. Call
/cloudrecognition/targetCollection/:tcId/generation
again with the ID
of your TargetCollection and the process will be started. Since this call is asynchronous you will receive the response immediately with a path in the Location-property in the header of the response. By calling the url with the path, for example
/cloudrecognition/targetCollection/:tcId/generation/wtc/:generationId
with a GET-method request, you will see the status of the progress of the cloud archive generation in the response body as a JSON object. When the generation is completed, the cloud archive is available for recognition. Note that the generation process can take a while when generating a large TargetCollection for the first time. Small additions to existing cloud archives are processed a lot faster.
- your newly added image will not be recognized
- your deleted image will still be recognized
Your Cloud Archive is now ready on the server and can be used in combination with the Wikitude SDK from your app. See the SDK sample called Cloud Recognition for more details.
Generate a WTC file
You can create and download a wtc file of a specific Target Collection (Generate WTC) with up to 1000 targets by calling
/cloudrecognition/targetCollection/:tcId/generation/wtc
with method POST. You have to specify the SDK version the wtc file should be built for in the request body. Valid values for the version are "3.x", "4.0", "4.1", and "5.0". Optionally, an email address can be added. The email is used for a notification once the generation of the wtc file has finished. Example for the request body:
{
"sdkVersion": "5.0",
"iwantmywtcfile@wikitude-user.com"
}
Similar to the cloud archive generation this call is asynchronous, so the response header (Location) contains a path useful for requesting the status of the wtc creation. Once the status is COMPLETED
the link to the actual wtc file can be requested from the TargetCollection. The received TargetCollection
object (in the body of the response) contains an additional property called wtc
, which is an array of wtc objects. Those objects consists of the following properties:
- the
url
to the wtc file, - the number of targets (
nrOfTargets
), - the
version
, - the creation date (
creDat
)
Additional calls
Beside the above described steps the Manager API also offers to Delete TargetCollections and Delete Targets.
Using GET
request you can query details about a single TargetCollection, all Target Collections in your account, a single Target and all Targets within a TargetCollection.
The physical height and the metadata of an existing target can be updated.
Migrate
Migration notes for the Wikitude SDK Native API (iOS)
Migrate from 1.3 to 1.4
iOS 10 now requires a camera usage description. Please make sure that you add the NSCameraUsageDescription
key to your applications .plist.
WTWikitudeNativeSDK
- new
-clearCache
. Use this method to clear all internal SDK caches.
- new
Migrate from 1.0 to 1.1
-
- new
-create2DClientTrackerFromURL:extendedTargets:andDelegate:
Use this method to create a newWTClientTracker
object which is configured for 2D tracking. new
-create3DClientTrackerFromURL:andDelegate:
Use this method to create a newWTClientTracker
object that is configured for 3D tracking.deprecated
-createClientTrackerFromURL:extendedTargets:andDelegate:
Use-create2DClientTrackerFromURL:extendedTargets:andDelegate:
instead.
- new
new
WTTrackingMapRecorder
Use this class to record and save so called 'Tracking Maps' that are needed for 3D tracking.
Reference
iOS Native API Reference
Go to iOS Reference for a complete reference of all iOS Wikitude Native API objects and functions.
Cloud Recognition Manager API
Go to REST API Reference for a complete reference of all REST API calls for the Manager API.
Wikitude SDK iOS Native API Release Notes
Wikitude SDK Native API 1.4.0
Release Date: 13.09.2016
New
- Support for iOS 10
- New InputFrameRenderSettings options for Plugins API
Fixed
SDK
- Fixes issues with NAT64 networks
- Fixes a rare crash in the 3D Tracking engine
Improved
- Improved and aligned behaviour of cache handling
- Improved and aligned behaviour of network handling for cloud recognition
Wikitude SDK Native API 1.3.0
Release Date: 08.07.2016
New
- InputPlugins allow developer to use the Wikitude SDK with an external camera feed
Fixed
- Fixes an issue where destroying the SDK while a large .wtc file is loading, crashed the SDK
- Fixes several issues that might lead to a crash if the SDK is paused/resumed
- Fixes several issues that might lead to a crash if the SDK is destroyed and immediately afterwards created again
- Fixes an issue where green camera frames could be rendered under certain circumstances
- Fixes several issues where destroying the SDK might not release all of its allocated memory
Wikitude SDK Native API 1.2.1
Release Date: 22.03.2016 (internal release only)
Fixed
- Unifies camera related API between Android and iOS
Wikitude SDK Native API 1.2.0
Release Date: 01.12.2015
New
- Support multiple regional co-located cloud recognition services
- Improvements in visualization of 3D Tracking Beta
Wikitude SDK Native API 1.1.0
Release Date: 15.10.2015
New
- 3D Tracking Beta for small-sized objects
- Map recorder in sample App
- Updated Client Tracking sample with new 3D Tracking sample
Wikitude SDK Native API 1.0.0
Release Date: 28.08.2015
Wikitude SDK Native API 1.0.0 beta
Release Date: 30.07.2015
New
- Initial Public Release of iOS Native API