Getting started
Welcome to the Wikitude SDK. This document is designed to help you from your very first steps with the Wikitude SDK all the way through to advanced concepts and examples for developing your augmented reality project.
Recommended Usage of this Documentation
The documentation is arranged in a way to guide you through the various steps in your development process. We recommend following each of the steps outlined below and reading the documentation in the order displayed.
Setup your project
In this section we describe the necessary steps to setup a project in a detailed guide.
View the samples
All of the included samples are complete iOS projects and apps. Browse through this section and get an idea of what the SDK is capable of. The relevant parts of the samples are described in more detail to highlight the applied concepts and patterns. These examples are designed to help you get off to a great start with the Wikitude SDK.
Viewing samples based on vision based augmented reality requires the corresponding reference images. All of them are available directly in the description of the sample or available as a collection on this page, which you can either view on the screen or print.
The Wikitude SDK - Augmented Reality for your own app
The Wikitude SDK is a software library and framework for mobile apps used to create augmented reality experiences.
As opposed to the Wikitude SDK - JavaScript API where you define your AR experiences with JavaScript, you will create your AR experiences with ObjC and OpenGL when using the Wikitude Native SDK.
Architecture of the Wikitude SDK
The image above shows the different components of the Wikitude SDK and possible approaches for creating augmented reality apps. Each of these approaches is based on a certain development environments (IDE) and platforms:
- Computer Vision Engine: The computer vision engine is a core component of the Wikitude SDK and used by all platforms. It is not directly accessible, but wrapped either by the Native API or the JavaScript API.
- Wikitude SDK - Native API: Provides access to the Wikitude computer vision engine natively for Android (Java) and iOS (ObjC). It also can load plugins via the Wikitude Plugins API. (NOTE: Wikitude SDK plugins have nothing to do with the Cordova or Unity Plugin concept.)
- Wikitude SDK - JavaScript API: Allows to build augmented reality worlds on basis of HTML and JavaScript. It is available for Android and iOS. The JavaScript API provides access to the functionality of the computer vision engine, location based AR, the Plugins API and dedicated rendering functionality.
- Wikitude SDK - Plugins API: An API to connect your own plugins to the Wikitude SDK.
- Wikitude SDK - Cordova Plugin: On top of the JavaScript API the Cordova plugin allows to use the Wikitude SDK in combination with Apache Cordova.
- Wikitude SDK - Titanium Module: On top of the JavaScript API the Titanium module allows to use the Wikitude SDK in combination with Titanium.
- Wikitude SDK - Unity Plugin: On top of the Native API the Unity plugin allows to use the Wikitude SDK in combination with Unity.
- Wikitude SDK - Xamarin Component: On top of the JavaScript API the Cordova plugin allows to use the Wikitude SDK in combination with Xamarin.
The Wikitude Developer Portal
The Wikitude Developer Section should be your first stop when you have specific development related questions. The portal hosts a very active Developer Community Forum where Wikitude staff members are constantly assisting other developers with helpful tips and advice. A Knowledge Base helps with various questions.
Feedback and Contact
We are always interested in your feedback and suggestions how we can improve this documentation. Please use the contact form on our website or visit us on Google+ or Facebook
Setup Guide iOS Native API
There are only a few steps necessary to add the Wikitude Native SDK to your existing iOS application. This guide will explain them in detail. In general the steps are
- Adding the Wikitude Native SDK .framework to a Xcode project
- Using the main classes to create your augmented reality experiences
Project Setup
Load your Xcode project
The first step is to open an exiting Xcode project. If there is no project already created, do so using the Xcode project configurator.
Adding the Wikitude SDK Framework
To keep things clear, you should copy the Wikitude Native SDK .framework into your Xcode project structure. Having it somewhere else on your machine might lead to confusion when updating to a newer version of the Wikitude Native SDK.
After the .framework was copied, it can be added as 'Embedded Binaries'.
Because the Wikitude Native SDK will fully support Swift, a build setting has to be changed in order to package Swift Standard Libraries into the final application package.
The project is now fully configured to use the Wikitude Native SDK.
Using the Wikitude Native SDK in your Application
After the setup steps are completed, the Wikitude Native SDK is ready to be used within an iOS application.
The Wikitude Native SDK comes with an umbrella header that has to be included in your corresponding source files. The file is called WikitudeNativeSDK.h
and can be imported by calling #import <WikitudeNativeSDK/WikitudeNativeSDK.h>
The main class to work with is WTWikitudeNativeSDK
. It's the central point for creating client or cloud tracker, to obtain an already configured rendering component or to control camera related settings.
The Wikitude Native SDK can be used in trial mode without any license, but to remove the watermark, a valid license has to be set using the -setLicenseKey
method.
Application Lifecycle
To integrate the Wikitude Native SDK into the application lifecycle, use the -start:completion
and -stop
methods of WTWikitudeNativeSDK
. A good place to call them is e.g. a UIViewControllers
-viewWillAppear:
and -viewWillDisappear:
.
To refine the startup phase, the WTStartupConfiguration
object can be used which is the first arguments in first block parameter of -start:completion:
. The second block parameter can be used to check if the Wikitude Native SDK could actually start with the given configuration on the current device. Use the isRunning
parameter to determine the run state and e.g. create tracker in this block. The following listening demonstrates this:
[self.wikitudeSDK start:nil completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
NSURL *clientTrackerURL = [[NSBundle mainBundle] URLForResource:@"magazine" withExtension:@"wtc" subdirectory:@"Assets"];
self.clientTracker = [self.wikitudeSDK.trackerManager createClientTrackerFromURL:clientTrackerURL extendedTargets:nil andDelegate:self];
}
}];
Error Handling
The Wikitude Native SDK calls the WTWikitudeNativeSDK
delegate method -wikitudeNativeSDK: didEncounterInternalError:
as soon as an internally inconsistent state was detected and parts of the SDK might stop working as expected. This is especially helpful while developing the application and not so much for end user notifications.
Client/Cloud Tracker, Camera and Rendering
How client/cloud tracker, the camera controls and rendering can be used, will be explained in the example part of this documentation. Each feature will be explained in detail through a real usage example.
Supported Devices iOS
Wikitude SDK is running on devices fulfilling the following requirements:
Requirements for other operating systems and platforms are listed in this overview.
Supported Devices
Wikitude SDK is running on devices fulfilling the following requirements:
Sensor-based AR (Geo-AR) | Image recognition and tracking | |
---|---|---|
Android |
|
|
Epson |
|
|
Google Glass |
|
|
iOS |
|
|
Vuzix |
|
|
How to obtain a free trial license
The Wikitude SDK requires a valid license key to be able to run properly. An empty or missing license key will block the augmented reality view from showing any meaningful content. You will see a watermark across the screen with the words License Key Missing
. All JavaScript API calls will be ignored and not interpreted.
When downloading the Wikitude SDK you will be forwarded to the license generation page, where a trial license key is automatically generated for you.
Copy the key into your app, which will unlock the trial mode of the Wikitude SDK. The trial mode of the Wikitude SDK contains the full feature set of the Wikitude SDK but will show a Trial
watermark across the screen.
Each trial license key is valid for every application ID on every operating system. You can use the same trial license key in multiple apps.
Where should I enter the license key
iOS Native API
To use the Wikitude iOS SDK with a certain license key, use the method -setLicenseKey:
defined in WTWikitudeNativeSDK.h
and provide a valid license key.
Examples
The following examples should give you an overview of the capabilities offered by the Wikitude SDK. Each sample is capable of running without modifications on all supported platforms.
The Sample App is a fully functional sample project either for Android or iOS. You can easily import it into Android Studio or Xcode and work from there. The user interface is kept very simple and shows a list to select the sample you are interested.
Examples Application
This section describes the SDKExamples
application in detail and highlights the main features and use-cases of the Wikitude SDK.
Please note that the Xcode projects deployment target is set to iOS 8 and the application has to be deployed to a actual iOS device and not the simulator.
Structure
The Native Examples
application is structured as a table view where each cell launches an individual example, describing a specific feature of the SDK.
The application uses an UINavigationViewController
and pushes each example with a dedicated view controller who implements a complete use case. All view controller are grouped together in the Controller
group. Example specific assets are locates in the Assets
group. Rendering related classes that are used by multiple examples can be found in CoreServices
and Views
.
Usage
To run the application, open the Xcode project and click Run
.
Client Recognition
This example shows how to recognize images in the viewfinder and overlay it with images.
For a better understanding, here are some terms that will be used in the following and other section of this documentation related to vision-based augmented reality.
Target: A target image and its associated extracted data that is used by the tracker to recognize an image.
Target collection: An archive storing a collection of targets that can be recognized by the tracker. A target collection can hold up to 1000 targets.
ClientTracker: The tracker analyzes the live camera image and detects the targets stored in its associated target collection. Multiple trackers can be created, however only one tracker can be active for recognition at any given time.
Simple Client Tracking iOS
In this section we will go through the WTSimpleClientTrackerViewController code, which you can find in the example applications folder Native Examples/Controller/ClientTracking
. We will discuss general concepts on how to use the Wikitude Native SDK as we go along, please don't skip this sample even if you are only interested in cloud recognition for example.
Please refer to the Setup Guide section of this documentation to find detailed information on how to integrate the Wikitude Native SDK into a Xcode project.
In this example we will use external rendering, for details on how to setup the rendering and the difference between internal and external rendering please read through the section on rendering.
The WTWikitudeNativeSDK
class is structured to be used in a standard iOS view controller as a model of the MVC pattern. This class is central starting point for anything related to the Wikitude Native SDK. It provides factory methods to create different kinds of tracker and to receive information of those through there delegate objects.
A typical usage would be to have a strong reference to a WTWikitudeNativeSDK
object in a view controller and to define this view controller to be conform to the WTWikitudeNativeSDKDelegate
protocol
@interface WTSimpleClientTrackerViewController () <WTWikitudeNativeSDKDelegate, WTClientTrackerDelegate>
The WTWikitudeNativeSDKDelegate
protocol defines methods that can be used to react on SDK specific events. It is the chosen way to communicate e.g. runtime errors that can not be returned directly from an API call. Such errors might contain information about camera authorization status changes and inconsistent rendering setups.
To receive client tracker information about recognized/tracked or lost targets, this view controller is also conform to the WTClientTrackerDelegate
protocol. All methods in the mentioned protocol are optionals thus only provide information from the SDK but don't require any action from the application.
Additionally this protocol provides information about the loading status of a particular client tracker object.
The next step we will take is create an instance of the WTWikitudeNativeSDK
class. This is the only object that you have to create manually. All other object will be created by factory methods of the WTWikitudeNativeSDK object.
WTWikitudeNativeSDK
manually. All object somehow play together in a well defined manner and that why they are internally controlled by the WTWikitudeNativeSDK
object and its sub components like WTTrackerManger
and WTCaptureDeviceManager
.
A WTWikitudeNativeSDK
object can be instantiated at any time and does not depend on any particular application lifecycle event. To integrate internal processes into the hosting application, -start:completion
and -stop
are used. These two methods should either be called from - viewWillAppear: | - viewWillDisappear:
or -viewDidAppear: | -viewDidDisappear:
.
To refine the startup behaviour of the Wikitude Native SDK, the first block parameter of -start:completion:
contains an already created and setup object of type WTStartupConfiguration
. Public properties of this object can be used to further define how the Wikitude Native SDK should start e.g. its camera. Typical settings from this object are the camera preset which defines in which resolution the device camera should be started or what's the desired target rendering frame rate.
The second block parameter contains a BOOL indicating if the given properties could be taken into account during the final startup phase. If an error occurred, isRunning
is set to NO
and the given NSError
object contains more information about what went wrong.
The second block parameter is also the ideal starting point to create any tracker object from the Wikitude Native SDK if the running state is YES.
To load client tracker objects, a target collection is needed. These target collections are represented through a .wtc file (Wikitude Target Collection). The .wtc file can be packed into the application bundle and a file URL retrieved using the NSBundle
method -URLForResource:withExtension:subdirectory:
or uploaded to a server and downloaded from the Wikitude Native SDK.
To get notified when the client tracker finished loading, its delegate method -clientTracker:didFinishedLoadingTargetCollectionFromURL:
is called or -clientTracker:didFailToLoadTargetCollectionFromURL:withError:
if the .wtc could not be loaded.
[self.wikitudeSDK start:^(WTStartupConfiguration *startupConfiguration) {
startupConfiguration.captureDevicePosition = AVCaptureDevicePositionFront;
} completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
NSURL *clientTrackerURL = [[NSBundle mainBundle] URLForResource:@"magazine" withExtension:@"wtc" subdirectory:@"Assets"];
self.clientTracker = [self.wikitudeSDK.trackerManager createClientTrackerFromURL:clientTrackerURL extendedTargets:nil andDelegate:self];
}
}];
Next we will have a closer look on how to get on tracker specific data. The WTClientTrackerDelegate
provides three methods that will be called when the tracker found, lost or tracked targets.
-baseTracker:didRecognizedTarget:
is called if a new target was found in the current camera frame-baseTracker:didTrackedTarget:
is called when an already known target moved to a new position-baseTracker:didLostTarget:
is called when an already known target could not be found anymore in the latest camera frame
The WTImageTarget object provides information about which target changed and what changed for the target, e.g. its position which would be represented through a OpenGL like model view matrix.
- (void)baseTracker:(nonnull WTBaseTracker *)baseTracker didRecognizedTarget:(nonnull WTImageTarget *)recognizedTarget
{
NSLog(@"recognized target '%@'", [recognizedTarget name]);
_isTracking = YES;
}
- (void)baseTrakcer:(nonnull WTBaseTracker *)baseTracker didTrackTarget:(nonnull WTImageTarget *)trackedTarget
{
[self.renderableRectangle setProjectionMatrix:trackedTarget.projection];
[self.renderableRectangle setModelViewMatrix:trackedTarget.modelView];
}
- (void)baseTracker:(nonnull WTBaseTracker *)baseTracker didLostTarget:(nonnull WTImageTarget *)lostTarget
{
NSLog(@"lost target '%@'", [lostTarget name]);
_isTracking = NO;
}
- (void)clientTracker:(nonnull WTClientTracker *)clientTracker didFinishedLoadingTargetCollectionFromURL:(nonnull NSURL *)URL
{
NSLog(@"Client tracker loaded");
}
- (void)clientTracker:(nonnull WTClientTracker *)clientTracker didFailToLoadTargetCollectionFromURL:(nonnull NSURL *)URL withError:(nonnull NSError *)error
{
NSLog(@"Unable to load client tracker. Reason: %@", [error localizedDescription]);
}
Extended Client Tracking iOS
Based on the above to enable Extended Tracking for a tracker you need to provide a String array which defines which targets should be extended. In the sample we simply set a wildcard *
so that all targets in this tracker are extended.
self.clientTracker = [self.wikitudeSDK.trackerManager createClientTrackerFromURL:clientTrackerURL extendedTargets:@[@"*"] andDelegate:self];
Cloud Recognition
This example shows how to recognize images on a cloud server and then overlay it with augmentations utilizing the CloudTracker
class.
For a better understanding, here are some terms that will be used in the following and other sections of this documentation related to vision-based augmented reality.
Target: An image and its associated extracted data that is used to recognize an image.
Target Collection: A group of
targets
that are searched together. Think of it as a directory, which contains all your images you want to search. The Wikitude SDK can work with two different sorts ofTarget Collections
- On-device Target Collection: a static
wtc
file containing the extracted data of your images. Can consist of up to 1,000 images. - Cloud Target Collection: A target collection stored on the Wikitude server. See
Cloud Archive
below. Can consist of up to 50,000 images.
- On-device Target Collection: a static
Cloud Archive: An archive stored on the server that is optimized for cloud-based recognition. It is generated from a
Target Collection
and is used in combination withCloudTracker
.CloudTracker: Instead of analysing and computing the live camera feed directly on the device like the
ClientTracker
, theCloudTracker
will send the image(s) taken by the camera to the Wikitude Cloud Recognition server. The server will then do the hard work of trying to match the image with your targets in the specified cloud archive. Beside the benefit of searching in large image database using theCloudTracker
instead ofClientTracker
has also a positive impact on the general performance in most cases. Especially when using a large target collection and on older devices.
For both Cloud Recognition samples below we will use external rendering if you don't know what that means please go through the section on rendering before starting here.
The CloudTracker
is able to run in two modes, we call them on-click and continuous. In On-Click mode a single recognition cycle will be executed, in continuous mode the recognition will be started in a variable interval.
Both view controller are located in Controller/CloudTracking
.
On-Click Cloud Tracking
The starting point for on-click recognition is WTOnClickCloudRecognition
. In -viewDidLoad
we create a new instance of WTWikitudeNativeSDK
and initialize it with a valid license key.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
[self.wikitudeSDK setLicenseKey:kWTLicenseKey];
Then in -viewDidAppear:
the Wikitude Native SDK is started using the -start:completion:
method. If the SDK could be started, a object of type WTCloudTracker
is created using the WTTrackerManager
factory method -createCloudTrackerWithToken:targetCollectionId:extendedTargets:andDelegate
. The returned pointer needs to be retained in order to keep it alive, so assign it to a strong property.
[self.wikitudeSDK start:nil completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
NSLog(@"Wikitude SDK is not running. Reason: %@", [error localizedDescription]);
}
else
{
self.cloudTracker = [self.wikitudeSDK.trackerManager createCloudTrackerWithToken:@"b277eeadc6183ab57a83b07682b3ceba" targetCollectionId:@"54e4b9fe6134bb74351b2aa3" extendedTargets:nil andDelegate:self];
}
}];
Creating a cloud tracker object will automatically load it in the background. To get notified when the cloud tracker finished loading, implement the WTCloudTrackerDelegate
protocol. It provides two methods that are called once the cloud tracker finished loading or failed to load.
- (void)cloudTrackerFinishedLoading:(WTCloudTracker * __nonnull)cloudTracker
{
NSLog(@"cloud tracker finished loading");
}
- (void)cloudTracker:(WTCloudTracker * __nonnull)cloudTracker failedToLoadWithError:(NSError *)error
{
NSLog(@"Cloud tracker failed to load with error: %@", [error localizedDescription]);
}
To start a single recognition, call the cloud tracker -recognize:errorHandler:
method. This method takes two block objects as parameter. The first on is called if a successful server connection could be established and the cloud tracker received a valid server response. The second one is called if any kind of error occurred during this communication. To call the -recognize:errorHandler:
method, a UIButton
is set up to call a specific method once it was touched.
- (IBAction)sendCloudRecognitionRequest:(id)sender
{
if ( self.cloudTracker )
{
if (self.cloudTracker.isLoaded)
{
[self.cloudTracker recognize:^(WTCloudRecognitionResponse *response) {
if ( response.recognized )
{
NSLog(@"Recognized target '%@' which has a rating of '%ld'", [response.targetInformations objectForKey:WTCloudTrackerResponseKey_TargetName], (long)[[response.targetInformations objectForKey:WTCloudTrackerResponseKey_TargetRating] integerValue]);
NSLog(@"Associated (custom) metadata: %@", response.metadata);
}
else
{
NSLog(@"No target image found in the analysed camera frame.\nKeep Calm \n\t&\nKeep Looking");
}
} errorHandler:^(NSError *error) {
NSLog(@"Cloud recognition recognition failed. Reason: %@", [error localizedDescription]);
}];
}
else
{
NSLog(@"Cloud tracker is not yet loaded. No request send.");
}
}
}
Please note that only a recognize call is made if the cloud tracker finished loading.
In case the server communication was successful, the success handler is called with an object of type WTCloudRecognitionResponse
that represents the server result. This object contains information if a image target was found or not, its name and, if specified, its associated meta data.
In case of an error, a NSError
object is passed to the errorHandler which contains detailed information about why the communication failed.
Since the WTCloudTrackerDelegate
protocol also conforms to WTBaseTrackerDelegate
, all tracking related methods to draw a augmentation around the recognized target are available as well.
Continuous Cloud Tracking
On-Click recognition is useful in some particular cases, but more often than not you probably want to use continuous recognition. For continuous cloud recognition we set an interval in which the CloudTracker
automatically calls the recognize function.
To start continuous recognition, simply call the -startContinuousRecognitionWithInterval:successHandler:interruptionHandler:errorHandler:
cloud tracker method once it finished loading.
if ( self.cloudTracker && self.cloudTracker.isLoaded )
{
[self.cloudTracker startContinuousRecognitionWithInterval:1.5 successHandler:^(WTCloudRecognitionResponse *response) {
NSLog(@"received continuous response...");
if ( response.recognized ) {
dispatch_async(dispatch_get_main_queue(), ^{
[self.cloudTracker stopContinuousRecognition];
[self.continuousRecognitionButton setTitle:@"Start Continuous Recognition" forState:UIControlStateNormal];
});
}
} interruptionHandler:nil errorHandler:^(NSError *error) {
}];
[self.continuousRecognitionButton setTitle:@"Stop Continuous Recognition" forState:UIControlStateNormal];
}
else
{
NSLog(@"Cloud tracker is not ready yet.");
}
As before, this method is called once a UIButton
was touched.
The first parameter defines in which interval new camera frames should be sent to the server for processing. The second parameter is again a success handler which contains information about a recognized target. The third parameter is a block that is invoked if the given interval is to short for the server to respond. The last parameter is again a error handle which is invoked in case of any error the occurred during the communication.
To stop a continuous recognition session, e.g. once a target was found, simply call the -stopContinuousRecognition
method.
Rendering
This example shows and explain how rendering works in combination with the Wikitude SDK Native API. There are two methods of rendering available in the Wikitude Native SDK. We call them internal and external rendering. Internal means the OpenGL view is setup by the Wikitude SDK and the SDK user can define custom rendering, that is executed by the Wikitude SDK. On the other hand external rendering means the SDK user sets up the OpenGL view and integrates the Wikitude SDK into this rendering setup.
External Rendering
Most of the examples in the Wikitude Native SDK example application use external rendering, simply because it might be the preferred usage by developers. The example application comes with a simple OpenGL ES 2 view (ExternalEAGLView
) and an also very simple renderer (ExternalRenderer
).
To activate external rendering you need to pass WTRenderingMode_External
to the WTWikitudeNativeSDK
method -initWithRenderingMode:delegate
.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
During the Wikitude Native SDK startup phase, which is initialized with a call to -start:completion:
, several methods from the WTWikitudeNativeSDKDelegate
protocol are called. They are needed to prepare the Wikitude Native SDK for external rendering. All methods in the WTWikitudeNativeSDKDelegate
protocol are marked as optional, but the Wikitude Native SDK will call its -wikitudeNativeSDK:didEncounterInternalError:
delegate method if a required method is not implemented or returned an invalid value. The required methods are:
- -wikitudeNativeSDK:didCreatedExternalUpdateHandler:
- -wikitudeNativeSDK:didCreatedExternalDrawHandler:
- -eaglViewSizeForExternalRenderingInWikitudeNativeSDK:
- -eaglContextForVideoCameraInWikitudeNativeSDK:
- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreatedExternalUpdateHandler:(WTWikitudeUpdateHandler __nonnull)updateHandler
{
self.wikitudeUpdateHandler = updateHandler;
}
- (void)wikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK didCreatedExternalDrawHandler:(WTWikitudeDrawHandler __nonnull)drawHandler
{
self.wikitudeDrawHandler = drawHandler;
}
- (CGRect)eaglViewSizeForExternalRenderingInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
return self.eaglView.bounds;
}
- (EAGLContext *)eaglContextForVideoCameraInWikitudeNativeSDK:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
if (!_sharedWikitudeEAGLCameraContext )
{
EAGLContext *rendererContext = [self.renderer internalContext];
self.sharedWikitudeEAGLCameraContext = [[EAGLContext alloc] initWithAPI:[rendererContext API] sharegroup:[rendererContext sharegroup]];
}
return self.sharedWikitudeEAGLCameraContext;
}
The first two methods provide block objects, that need to be called every frame from the external rendering system. There purpose is to update internal SDK logic and draw the camera stream in OpenGL.
The last one is used to get to know the current OpenGL view size in order to draw the camera frame fullscreen and with the correct aspect ratio.
External rendering also requires a shared EAGLContext
object that is used by the Wikitude Native SDK to issue camera related OpenGL ES 2 calls. This way custom rendering and Wikitude rendering do not influence each other and less conflicts can occur.
An example of how those block objects can be used is the following:
if ( self.wikitudeUpdateHandler
&&
self.wikitudeDrawHandler )
{
self.wikitudeUpdateHandler();
self.wikitudeDrawHandler();
}
// ...external rendering code...
Such a snippet is typically found somewhere in the external render loop.
Internal Rendering
In case no OpenGL ES 2 rendering is already setup in the hosting Wikitude Native SDK application, the Wikitude Native SDK brings its own set of OpenGL ES 2 compatible classes to do so. These classes are WTEAGLView
and WTRenderer
. Objects of those classes can be retrieved through WTWikitudeNativeSDK
methods. To start the Wikitude Native SDK with internal rendering, pass WTRenderingMode_Internal
to the -initWithRenderingMode:delegate:
method.
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_Internal delegate:self];
self.wikitudeRenderer = [self.wikitudeSDK createRenderer];
self.wikitudeEAGLView = [self.wikitudeSDK createEAGLView];
After all required objects are created, view and renderer have to be connected through a call to setRenderer:
.
[self.wikitudeEAGLView setRenderer:self.wikitudeRenderer];
After those two objects are connected, the view has to be added to the view hierarchy of your application. This can either be done using Storyboards or programmatically. The Wikitude Native SDK example application does it in code.
[self.view addSubview:self.wikitudeEAGLView];
[self.wikitudeEAGLView setTranslatesAutoresizingMaskIntoConstraints:NO];
NSDictionary *views = NSDictionaryOfVariableBindings(_wikitudeEAGLView);
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"|[_wikitudeEAGLView]|" options:0 metrics:nil views:views]];
[self.view addConstraints:[NSLayoutConstraint constraintsWithVisualFormat:@"V:|[_wikitudeEAGLView]|" options:0 metrics:nil views:views]];
To do some custom OpenGL ES 2 calls from outside the Wikitude Native SDK, custom update and draw block objects can be passed to the Wikitude Native SDK. The corresponding delegate methods are called during the startup phase.
- (WTCustomUpdateHandler)wikitudeNativeSDKNeedsExternalUpdateHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
/* Intentionally returning a nil handler here */
return ^(){};
}
- (WTCustomDrawHandler)wikitudeNativeSDKNeedsExternalDrawHandler:(WTWikitudeNativeSDK * __nonnull)wikitudeNativeSDK
{
return ^(){
if ( _isTracking ) {
[self.renderableRectangle drawInContext:[EAGLContext currentContext]];
}
};
}
In this snippet only custom rendering is done and no update logic.
Plugins API
This guide consists of multiple sections, first we discuss Wikitude SDK Plugins in general, than we talk about platform specifics and how to register a plugin with the Wikitude SDK and then we go through each of the sample plugins included with the Wikitude Example Applications.
- About Wikitude SDK Plugins
- Platform Specifics
- Registering Plugins
- QR & Barcode Plugin
- Face Detection Plugin
About Wikitude SDK Plugins
Technically a plugin is a class, either written in C++ or Java, that is derived from the Wikitude Plugin base class. Beside lifecycle handling and options to enable and disable the plugin, the Plugin class has four main methods that you can override cameraFrameAvailable
which is called each time the camera has a new frame, update
which is called after each image recogntion cycle as well as 'startRender' and 'endRender' which are called before and after the Wikitude SDK does its rendering.
The most important thing to remember when working with plugins is that they need to have a unique identifier! If the attempt is made to register a plugin with an identifier that is already known to the Wikitude SDK, the register method call will return false.
Plugin Base Class
class Plugin {
public:
Plugin(std::string identifier_);
~Plugin();
string getIdentifier() const; // returns a unique plugin identifier
bool processesColorCameraFrames(); // returns true if the plugins wants to process color frames instead of bw
void setEnabled(bool enabled_);
bool isEnabled();
string callJavaScript(string javaScriptSnippet); // evaluates the given JavaScript snippet in the currently loaded ARchitect World context.
protected:
void initialize(); // called when the plugin is initially added to the Wikitude SDK
void pause(); // called when the Wikitude SDK is paused e.g. the application state changes from active to background
void resume(uint pausedTime_); // called when the Wikitude SDK resumes e.g. from background to active state. pausedTime represents the time in milliseconds that the plugin was not updated.
void destroy(); // called when the plugin is removed from the Wikitude SDK
void cameraFrameAvailable(const Frame&; cameraFrame_); // called each time the camera has a new frame
void update(const vector<RecognizedTarget> recognizedTargets_); // called each time the Wikitude SDK renders a new frame
void startRender(); // called before any Wikitude SDK internal rendering is done
void endRender(); // called right after any Wikitude SDK internal rendering is done
protected:
string _identifier;
bool _enabled;
};
With those methods in place your plugin will be able to read the full camera image for your own purpose, where the YUV image is also processed in wikitude’s computer vision engine.
Information about Recognized Targets
In case you have the wikitude SDK running with ongoing image recognition, the plugin API will populate the RecognizedTarget
in the update
method once an image has been recognized. The plugin can then work with class RecognizedTarget, which wraps the details of the target image in the camera view. With that you can read out the pose of the target image and use it for your purposes. Additionally the call contains the calculated distance to the recognized target
class RecognizedTarget {
public:
const string& getIdentifier() const; // the identifier of the target. The identifier is defined when the target is added to a target collection
const Mat4& getModelViewMatrix() const; // the model view matrix that defines the transformation of the target in the camera frame (translation, rotation, scale)
const Mat4& getProjectionMatrix() const;
const float getDistanceToCamera() const; // represents the distance from the target to the camera in millimeter
};
Passing values from within the plugin to the JavaScript part of your augmented reality experience is done via the addToJavaScriptQueue()
method of the Plugin class. Using this function will execute any JavaScript code in the context of your augmented reality experience.
Platform Specifics
Thanks to Objective-C++, C++ plugins can be loaded very easily on iOS. Simply create a new C++ file, include and derive from wikitude::sdk::Plugin
and create a std::shared_ptr
from your plugin class. The created shared pointer can then be supplied to the Wikitude SDK specific plugin registration/removal API.
To mark a Objective-C file as Objective-C++, change its file extension from .m to .mm or change the type manually to Objective-C++ using Xcodes Identity and Type inspector.
Registering Plugins
The Wikitude Native SDK for iOS offers three methods to register/remove C++ plugins. All of them are accessible through the WTWikitudeNativeSDK
class.
Registering a C++ Plugin
To register a C++ plugin, call the -registerPlugin:
method and pass a std::shared_ptr
wrapping your C++ plugin pointer. The following snippet shows how to define a property for a shared pointer, its initialization and the registration call.
@property (nonatomic, assign) std::shared_ptr<BarcodePlugin> barcodePlugin;
// ...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self); // arguments are passed to the C++ class constructor
// ...
[self.wikitudeSDK registerPlugin:_barcodePlugin];
Removing a C++ Plugin
To remove a already known C++ plugin, call either the -removePlugin:
or -removeNamedPlugin:
method. The first one takes a shared pointer as argument and tries to find a plugin that matches this plugins identifier. The second one tries to remove a C++ plugin based on a string parameter. The second one is interesting because it allows you to not have a property that holds the shared pointer, but to directly call std::make_shared
when calling -registerPlugin:
. If this is done, the hosting application has no reference to the plugin pointer anymore, but can still remove it using its unique identifier (which has to be known by the hosting application/developer).
[self.architectView removePlugin:_faceDetectionPlugin];
//...
[self.architectView removeNamedPlugin:@"com.wikitude.plugin.face_detection"];
Barcode and QR code reader
This samples shows a full implementation of the popular barcode library ZBar into the Wikitude SDK. As ZBar is licensed under LGPL2.1 this sample can also be used for other projects.
ZBar is an open source software suite for reading bar codes from various sources, such as video streams, image files and raw intensity sensors. It supports many popular symbologies (types of bar codes) including EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Interleaved 2 of 5 and QR Code.
The C++ barcode plugin is created when WTBarcodePluginViewController
loads its view object.
- (void)viewDidLoad
{
[super viewDidLoad];
//...
self.wikitudeSDK = [[WTWikitudeNativeSDK alloc] initWithRenderingMode:WTRenderingMode_External delegate:self];
//...
_barcodePlugin = std::make_shared<BarcodePlugin>(640, 480, self);
}
When WTBarcodePluginViewController
starts the Wikitude Native SDK using the -start:completion:
method, the C++ barcode plugin is registered using the -registerPlugin:
method in the completion block.
- (void)viewDidAppear:(BOOL)animated
{
[super viewDidAppear:animated];
//...
[self.wikitudeSDK start:^(WTStartupConfiguration *startupConfiguration) {
//...
} completion:^(BOOL isRunning, NSError * __nonnull error) {
if ( !isRunning ) {
//...
}
else
{
//...
[self.wikitudeSDK registerPlugin:_barcodePlugin];
}
}];
}
Please note that the plugin is only registered when the Wikitude Native SDK could actually start.
Now let's move to the plugins C++ code. First we'll have a look at the BarcodePlugin.h
file. To create the bar code plugin we derive our BarcodePlugin
class from wikitude::sdk::Plugin
and override initialize
, destroy
, cameraFrameAvailable
and update
. We also declare the following member variables: _worldNeedsUpdate
, _image
, _imageScanner
and _presentingViewController
. The _worldNeedsUpdate
variable will later be used as an indicator if we need to display the recognized barcode, _image
and _imageScanner
are classes from zBar
which we'll use to scan for bar codes and _presentingViewController
will hold a pointer to the WTBarcodePluginViewController
from which we call a method every time we recognize a new barcode or QR code.
#include <zbar.h>
#import <WikitudeNativeSDK/Plugin.h>
#import <WikitudeNativeSDK/Frame.h>
#import <WikitudeNativeSDK/RecognizedTarget.h>
@class WTBarcodePluginViewController;
class BarcodePlugin : public wikitude::sdk::Plugin {
public:
BarcodePlugin(int cameraFrameWidth, int cameraFrameHeight, WTBarcodePluginViewController *presentingViewController);
virtual ~BarcodePlugin();
virtual void initialize();
virtual void destroy();
virtual void cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_);
virtual void update(const std::list<wikitude::sdk::RecognizedTarget>& recognizedTargets_);
protected:
int _worldNeedsUpdate;
zbar::Image _image;
zbar::ImageScanner _imageScanner;
WTBarcodePluginViewController *_presentingViewController;
};
In the constructor we set _worldNeedsUpdate
to zero indicating that there is no update necessary and initialize the zBar::Image
member variable passing its constructor the width and height of the camera frame, the image format of Y800
, set its data pointer to null and the data length to zero.
In the initialize
method we configure the zbar::ImageScanner
by calling setConfig
, enabling all supported bar codes. If you are only interested in one or some particular types of codes, first disabling all bar code types and manually enabling each particular type would be the better idea. That way performance could be greatly improved.
void BarcodePlugin::initialize() {
_imageScanner.set_config(zbar::ZBAR_NONE, zbar::ZBAR_CFG_ENABLE, 1);
}
We react to the destroy
event by setting the current data pointer of the zbar::Image
member to null and length to zero.
void BarcodePlugin::destroy() {
_image.set_data(nullptr, 0);
}
The last but most interesting methods are cameraFrameAvailable
and update
. In the cameraFrameAvailable
method we set the data of our previously initialized zbar::Image
member variable to the frame data we just received and the length of the data to frame width * frame height by calling set_data
. We then start the scanning process by calling the scan
method of our zBar::ImageScanner
passing the zBar::Image
member instance. The zBar::ImageScanner::scan
method returns the number of detected bar codes in the image frame, we save this number in a local variable n
. If n
is not equal to the result of the last frame, which we saved to _worldNeedsUpdate
member variable, we know there was a new bar code detected (meaning there was no bar code in the last frame) or that there was a bar code in the last frame and now there isn't. When that's the case, we do another check if there really was a bar code detected this frame and if there was, we call the presentBarcodeResult
method, passing the code content.
void BarcodePlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
int frameWidth = cameraFrame_.getSize().width;
int frameHeight = cameraFrame_.getSize().height;
_image.set_data(cameraFrame_.getLuminanceData(), frameWidth * frameHeight);
int n = _imageScanner.scan(_image);
if ( n != _worldNeedsUpdate ) {
if ( n ) {
zbar::Image::SymbolIterator symbol = _image.symbol_begin();
[_presentingViewController presentBarCodeResult:[NSString stringWithUTF8String:std::string(symbol->get_data()).c_str()]];
}
}
_worldNeedsUpdate = n;
}
Back in the Objective-C class WTBarcodePluginViewController
, the -presentBarCodeResult:
method is implemented like the following:
- (void)presentBarCodeResult:(NSString *)barcodeScanResult
{
if ( barcodeScanResult ) {
dispatch_async(dispatch_get_main_queue(), ^{
[self.barcodeLabel setText:barcodeScanResult];
});
}
}
Please note that the -presentBarcodeResult:
method is not called from the main thread, meaning that we need to schedule any UI related calls on the main thread. This is done by executing a block asynchronously on the dispatch main queue.
To visualize the detected bar code, we use a NSLabel and set its text property to the detected bar code string.
Face Detection
This samples shows how to add face detection to your Wikitude augmented reality experience using OpenCV.
The Face detection plugin example consists of the C++ classes FaceDetectionPlugin
, FaceDetectionPluginConnector
and the Objective-C class WTFaceDetectionPluginViewController
. We will use OpenCV to detect faces in the current camera frame and OpenGL to render a rectangle around detected faces.
FaceDetectionPluginConnector
is used to connect the FaceDetectionPlugin
class with the WTFaceDetectionPluginViewController
. Since this class mainly exists to ease the implementation of a cross platform plugin, we will not go into any implementation details for this class. We also don't go into any OpenCV or OpenGL details. If one is interested in such topics, the source code is part of the Wikitude Native SDK example application.
WTFaceDetectionPluginViewController
handles the face detection plugin creation and registration exactly as described in the previously bar code example. New for this example is a class method that converts UIDeviceOrientations into an int that is used by the face detection plugin to rotate the camera frame in order to always have faces in the right orientation. This class method is then used every time the device changes its orientation.
__weak typeof(self) weakSelf = self;
[[NSNotificationCenter defaultCenter] addObserverForName:UIDeviceOrientationDidChangeNotification object:[UIDevice currentDevice] queue:[NSOperationQueue mainQueue] usingBlock:^(NSNotification *note) {
weakSelf.faceDetectionPlugin->setFlipFlag( [WTFaceDetectionPluginViewController flipFlagForDeviceOrientation:[[UIDevice currentDevice] orientation]] );
}];
Next we have a look at the FaceDetectionPlugin
class. Again we we will leave out implementation details and focus on how we use the plugin itself. In the cameraFrameAvailable
method we use OpenCV to detect faces in the current camera frame which the Wikitude SDK passes to the plugin. We call the observer which is an instance of the FaceDetectionPluginConnector
to notify the view controller about the result. The plugin base class defines startRender
and endRender
, depending on, if you would like to render on top of or below of all rendering the Wikitude SDK does, you choose one of them, or both to override. To render below all Wikitude rendering we choose startRender
and again call the FaceDetectionPluginConnector
instance which in turn calls the view controller. Since we do not react on the result of the Wikitude SDK image recognition we leave update
blank.
... ctor/dtor ...
void FaceDetectionPlugin::cameraFrameAvailable(const wikitude::sdk::Frame& cameraFrame_) {
wikitude::sdk::Size frameSize = cameraFrame_.getSize();
_scaledCameraFrameWidth = cameraFrame_.getScaledWidth();
_scaledCameraFrameHeight = cameraFrame_.getScaledHeight();
std::memcpy(_grayFrame.data,cameraFrame_.getLuminanceData(), frameSize.height*frameSize.width*sizeof(unsigned char));
//... Control Open CV ...
if ( _result.size() ) {
convertFaceRectToModelViewMatrix(croppedImg, _result.at(0));
_observer->faceDetected(_modelViewMatrix);
} else {
_observer->faceLost();
}
}
void FaceDetectionPlugin::startRender() {
_observer->renderDetectedFaceAugmentation();
}
void FaceDetectionPlugin::update(const std::list<wikitude::sdk::RecognizedTarget> &recognizedTargets_) {
/* Intentionally Left Blank */
}
//... other internally used methods ...
To render a frame around detected faces we created an instance of the StrokedRectangle
class which takes care of rendering a rectangle around faces and all targets of the also active ClientTracker
. When the plugin detects, looses or recalculated the projection matrix it will call the appropriate view controller methods which we use to update the StrokedRectangle
instance. If the Plugin decides it is time to render a frame around a detected face it will call -setFaceIsRecognized:atPosition:
. Since the plugin will only call this method in the startRender
method, we know the current thread is the OpenGL thread and are able to dispatch OpenGL calls.
- (void)setFaceIsRecognized:(BOOL)recognized atPosition:(const float*)modelViewMatrix
{
self.faceDetected = recognized;
if (recognized) {
[self.recognizedFaceRectangle setModelViewMatrix:modelViewMatrix];
}
}
- (void)setFaceAugmentationProjectionMatrix:(const float*)projectionMatrix
{
[self.recognizedFaceRectangle setProjectionMatrix:projectionMatrix];
}
Camera Controls
The WTCaptureDeviceManager
allows you to change capture device specific settings during an active capture session. It lets you change e.g.
* the capture device position
* the capture device focus mode
* the capture device zoom level
A valid pointer to a WTCaptureDeviceManager
object can be retrieved from the WTWikitudeNativeSDK
method -captureDeviceManager
. Please note that this pointer is only valid if the native SDK is running. If the SDK is currently stopped, changes for the capture device are done using the WTStartupConfiguration
object that is passed as argument to the startupHandler when calling -start:completion:
.
The sample uses a UIPickerView
to let the user select between different camera specific values. This UIPickerView
is added to the view hierarchy in the main storyboards camera scene. By default, the picker view is placed behind the OpenGL ES 2 view, so that it is not visible. To show the picker view, the bottom OpenGL ES 2 view constraint is modified and animated, if a button is touched, to unveil the picker view. A standard iOS SDK UIView animation is used to do so. Please refer to the WTCameraControlsViewController
-handleCameraControlsPresentation
method.
The connection between the UIPickerView
and WTCaptureDeviceManager
is done by a custom class called WTCameraControlsModel
and its delegate object (Which is in this case WTCameraControlsViewController
). This custom class WTCameraControlsModel
implements the UIPickerView
data source and delegate protocol and uses its custom delegate object to retrieve current camera specific values or exposes newly selected values from the picker view.
Implementation details of WTCameraControlsModel
are not discussed in this guide as it only handles UIKit related functionality. The following snippets shows the previously mentioned object connection. All following snippets can be found in the WTCameraControlsViewController
class.
self.cameraControlsPickerModel = [[WTCameraControlsModel alloc] init];
self.cameraControlsPickerModel.delegate = self;
self.cameraControlPicker.dataSource = _cameraControlsPickerModel;
self.cameraControlPicker.delegate = _cameraControlsPickerModel;
The camera controls model knows internally about all possible camera settings and uses these information to load the picker view. Once the picker is loaded and the user selects a different value, the camera controls model delegate is used to propagate the value change. Since the camera controls view controller implements this delegate protocol and also holds a reference to the Wikitude Native SDK, which by itself provides access to the capture device manager), the view controller sets the appropriate capture device value when this camera controls model delegate is called. The following snippet demonstrates this for the capture device position:
- (void)cameraControlsModel:(WTCameraControlsModel *)cameraControlsModel didSelectCameraPosition:(AVCaptureDevicePosition)activeCameraPosition
{
if ( [self.wikitudeSDK captureDeviceManager] ) {
[[self.wikitudeSDK captureDeviceManager] setActiveCaptureDevicePosition:activeCameraPosition];
}
}
Please note that there is a nil check when [self.wikitudeSDK captureDeviceManager]
is called. This is because the Wikitude Native SDK could be paused when the picker changes one of its values.
Every time the picker becomes visible again, the camera controls model asks through its delegate object which capture device settings are currently active. The following snippet again demonstrates this for the currently active capture device position.
- (AVCaptureDevicePosition)currentlyActiveCameraPositionForCameraControlsModel:(WTCameraControlsModel *)cameraControlsModel
{
return [[self.wikitudeSDK captureDeviceManager] activeCaptureDevicePosition];
}
Target Images
All samples
Click here to download all target images
Wikitude Cloud Recognition
The Wikitude Cloud Recognition service is cloud-based service provided by Wikitude, which recognizes images sent from Android and iOS apps using the Wikitude SDK. The recognized images are then tracked in the live camera feed and can be used for augmented reality experiences.
This documentation focuses on the RESTful API called Manager API, which is used to interact on a back-end level with the Cloud Recognition service.
General Definitions
Target: An image and its associated extracted data that is used to recognize an image.
Target Collection: A group of
targets
that are searched together. Think of it as a directory, which contains all your images you want to search. The Wikitude SDK can work with two different sorts ofTargetCollections
- On-device Target Collection: a static
wtc
file containing the extracted data of your images. Can consist of up to 1,000 images. - Cloud Target Collection: A target collection stored on the Wikitude server. See
Cloud Archive
below. Can consist of up to 50,000 images.
- On-device Target Collection: a static
Cloud Archive: An archive stored on the server that is optimized for cloud-based recognition. It is generated from a
TargetCollection
and is used in combination with the Wikitude SDKAR.CloudTracker
.
Manager API: A RESTful web API allowing developers to interact with the Cloud Recognition server for managing Targets
, TargetCollections
and Cloud Archives
. Only you as a developer uses this API. None of your users of your app will interact with this API.
Client API: The Client API is the interface between the Wikitude SDK and the Cloud Recognition Service. The API itself is encapsulated in the Wikitude SDK class AR.CloudTracker
and not directly accessible. Calls on the client API are called Scans
.
Getting Started with the Cloud Recognition Service
Preparation
First Steps and General Usage
- Get familiar with the Manager API calls in the API Reference
- Create a first Target Collection using the
Create Target Collection
endpoint and note down the ID of the Target Collection - Create targets using the
Create Target
endpoint for that particular Target Collection - Important:
Generate a Cloud Archive
for your Target Collection - Go to the Wikitude SDK and create an Android or iOS project
- Use the Client API token to authenticate your Android or iOS project
- Use your Target Collection ID to recognize images
For more information on the available endpoints and how to work with the Manager API see the workflow section.
Authentication
The Cloud Recognition Service knows two authentication tokens, that you need in order to work with the service
Manager API token You need this token to authenticate yourself against the RESTful Manager API. The Manager API is used to create, add and delete targets and target collections. The token identifies your developer account. Calls to the Manager API do not count towards your quota limits.
Client API token You need this token to authenticate calls from the Wikitude SDK to the Cloud Recognition services. It again authenticates calls as legitimate. The token is bound to your developer account. Calls from the Wikitude SDK to the service with a wrong or missing token can not access your target collections.
Authentication on the Manager API
The Manager API token must be added to each call towards the Wikitude Cloud Recognition Manager API. The token authenticates the user account that is using the API.
Authentication on the Client API
The Client API token must be added to your app project using the Wikitude SDK. This token is needed additionally beside the SDK license key when working with the AR.CloudTracker
class.
Quota and Limits
General Upload Limit
The Wikitude Cloud Recognition will not accept images bigger than 1024kB (1 MB). Trying to upload images exceeding this file size will result in a HTTP status code 400
together with an error message FILE_SIZE_LIMIT_EXCEED
.
Limits for the Wikitude Cloud Recognition Service
There are two main limitations for the Wikitude Cloud Recognition service that you need to be aware of:
Targets Your token has a certain number of targets that you can upload and store on the cloud service under your developer account. The limit is always counted for your entire developer account and not for a single target collection. The service is not counting single uploads, but how many targets are currently stored in target collections under your account.
Scans Scans are in effect calls from the Wikitude SDK via the Client API to the Cloud Recognition servers. All commercial license come with an allowance of 1,000,000 scans per month per developer account. Note: When using Continuous Search mode multiple calls are made to the server.
Maximum Number of Targets in a Target Collection
A target collection can't exceed 50,000 targets.
Free Trial License for Cloud Recognition
Wikitude provides a trial token for each developer account to try out the Cloud Recognition for free. This trial token has set a quota limit that allows developers to try and test the functionality of the service. Limitations for trial accounts
- Targets: 50,000
- Scans: 1,000 per month
To get your trial token, please visit the License page.
Commercial Licenses
For production systems, we offer commercial licenses with various quota limits for purchase.
Product | Targets | Scans |
---|---|---|
Cloud Recognition 1000 | 1,000 | 1,000,000 |
Cloud Recognition 10000 | 10,000 | 1,000,000 |
Cloud Recognition 25000 | 25,000 | 1,000,000 |
Cloud Recognition 50000 | 50,000 | 1,000,000 |
Cloud Recognition 100000 (*) | 100,000 | 1,000,000 |
(*) Maximum number of targets per target collection can't exceed 50,000
Your first Target Collections
Target Collections are central to working with Cloud Recognition service. They keep all your target images and are the base for the cloud archive.
Think of TargetCollection as a directory, where your images are stored. A TargetCollection forms a logical group, which is searched as a whole. Of course you can have several TargetCollections in your account, each consisting up to 50,000 images each.
What is the difference between Cloud Archive and Target Collection
Structure of a TargetCollection
Property | Type | Description |
---|---|---|
id | (String) | An ID that uniquely identifies the TargetCollection |
name | (String) | The Name of the TargetCollection, as defined by the user |
creDat | (Number) | A timestamp when the TargetCollection was created (as returned by JavaScript's Date.now() function) |
modDat | (Number) | A timestamp when the TargetCollection was last modified (as returned by JavaScript's Date.now() function) |
Create a Target Collection
Creating a Target Collection is easy and can be done without any prerequisites in your account. In general it is your starting point and most likely your very first action.
Call the endpoint
https://api.wikitude.com/cloudrecognition/targetCollection
with the mandatory name
field as a POST
request and you will create a new TargetCollection. The response will contain a TargetCollection object, where the ID is most important parameter. You can also add metadata
to a TargetCollection in case you want to some additional descriptive information. The next step is to add images to your TargetCollection, so they can be recognized.
Add Target Images
A Target is an plain image that can be recognized by the Wikitude Cloud Recognition service. Adding or creating a target means to provide a URL to your image to the server, which then downloads the image, analyzes it and adds it to the TargetCollection. To add an image call the endpoint
https://api.wikitude.com/cloudrecognition/targetCollection/:tcId/target
with the ID
of the TargetCollection, where you want to add the image. You need to add the a field imageUrl
to your request. The image must be publicly accessible.
Pay attention to the optional fields name
and metadata
. name
is a unique identifier for your target within the TargetCollection. It is up to you to set and use this. The same is true for the metadata
object, which takes a full JSON object and can be filled with any value you like. The metadata
object will be present in the recognition response.
Structure of a Target
Property | Type | Description |
---|---|---|
id | (String) | An ID that uniquely identifies the Target |
name | (String) | The Name of the Target, as defined by the user |
imageUrl | (String) | The URL pointing to the original, uncompressed and uncropped Target binary file |
thumbnailUrl | (String) | The URL pointing to a thumbnail representation of the Target |
rating | (Number) | The rating (from 0 to 3) of the Target |
fileSize | (Number) | The file size of the original Target binary image file, in KB |
creDat | (Number) | A timestamp when the Target was created (as returned by JavaScript's Date.now() function) |
modDat | (Number) | A timestamp when the Target was last modified (as returned by JavaScript's Date.now() function) |
metadata | (JSON) | Arbitrary JSON data that is stored together with the target. |
Generate a Cloud Archive
Once you are done with adding targets you need to tell the server that it should generate your TargetCollection into a Cloud Archive. Call
https://api.wikitude.com/cloudrecognition/targetCollection/:tcId/generation
again with the ID
of your TargetCollection and the process will be kicked-off. Note that the generation process can take a while when generating a large TargetCollection for the first time. Small additions to existing cloud archives are processed a lot faster. While a cloud archive is generated it is locked and you can't execute any other actions on it. A 204
HTTP response will tell you that the generation process was successful and is done. In case of a time-out you can call the endpoint again and it will tell you the status of the generation process - see the three different error codes for a 412
HTTP response.
- your newly added image will not be recognized
- your deleted image will still be recognized
Your Cloud Archive is now ready on the server and can be used in combination with the Wikitude SDK from your app. See the SDK sample called Cloud Recognition for more details.
Additional calls
Beside the above described steps the Manager API also offers to Delete TargetCollections and Delete Targets.
Using GET
request you can query details about a single TargetCollection, all Target Collections in your account, a single Target and all Targets within a TargetCollection.
Tools
In this section of the documentation we are covering tools shipped with the Wikitude SDK, which help you in your development.
Target Management
This guide gives you an overview of how to create a target collection that you can use to detect and track images within your augmented reality experience.
In general the conversion can be done via three different tools:
- Web Target Manager Tool: A browser based tool to convert your images to a wtc file. You can find the tool under: www.wikitude.com/developer/tools/target-manager. You need your free developer account to log-in. This tool is described in more detail on further below
- Targets API: A RESTful API to convert images.
- Targets Enterprise Script: A binary shell script available for Mac OS X and Linux converting images to target collections. Pleases contact Wikitude Sales team for technical requirements and pricing.
The following images describes the relationship between the above mentioned methods and the Wikitude Cloud Recognition Service, which is not scope of this documentation.
Web Target Manager
Add a new project
- Open http://www.wikitude.com/developer/tools/target-manager and login with your Wikitude Developer account
- Add a new project to your project collection
Add target images to your own project
- Enter the newly created project
- Add new target images to the project either by clicking on
Add images
or drag & drop them on the gray area. Supported file formats include PNG and JPEG. If you are using PNG images, please make sure that it does not contain any transparent pixels, only solid colored images are supported.
The file name will be set as the
target name
that will later be used to reference a particular target in your ARchitect World. If thetarget name
is not completely visible, hover over it to reveal the full name.The target management tool will take a couple of seconds to calculate how suitable the provided target images are for detection and tracking purposes and generate a 3-star rating for each of them. Hover over the star rating to get additional information.
If you add your own target images you need the target name to set them in
AR.Trackable2DObject
.Star rating explained
- 0 stars: Not suitable for tracking. This target image cannot be tracked because it lacks textured features with high local contrast. Please consider choosing another target image.
- 1 star: Limited tracking ability. This target image provides basic tracking performance in good lightning conditions. Please consider improving the image
- 2 stars: Good tracking ability. This target image will track well in most conditions.
- 3 stars: Very good tracking ability. This target image will track very well in most conditions.
General advice for reference images
- Good image characteristics:
- Diversely textured image with high local contrast
- Bad image characteristics:
- Large areas with solid color or smooth color transitions
- Repetitive patterns
- Logos, signs
Create a target collection
- Select all desired targets you want to recognize/track, enter a name for the target collection and click on Generate target collection. Depending on the number of selected targets, it can take a moment to generate the target collection.
- After the target collection has been generated, it will be listed in the target collections page. Here you can review all target collections you've created. Use the provided download link to load the target collection directly from the Wikitude server or download it to package it together with your application for offline usage.
Use the target collection in your ARchitect World
Look at one of the client recognition examples or refer to the JavaScript API reference of AR.ClientTracker
for instructions on how to use the created target collection for augmentations in your ARchitect Worlds.
Best practice for target images
This guide gives you an overview of how to create a target collection that you can use to detect and track images within your ARchitect World.
Summary
Preferred images have:
- between 500 to 1000 pixels in each dimension
- Rich contrast
- Evenly distributed textured areas
- Many corner like structures
Unsuitable images have:
- Smaller dimensions than 500 pixels
- Larger than 1000 pixels as they do not provide more accurate results
- Large amounts of text
- Many repetitive patterns
- Large single-colored areas
- Color contrast only e.g. green to red edge), because all images are processed as grayscale images
Optimal Image Dimensions
- Optimal images are sized between 500 and 1000 pixels in each dimension
- Small images do not contain enough graphical information to extract so called feature points. The uniqueness, amount and distribution of features points are the key indicators for good detection and tracking quality
- Larger images do not improve the tracking quality
Low contrast images
- Images with high local contrast and large amount of rich textured areas is best suited for reliable detection and tracking
- Color contrast only (i.e. green to red edge) appears as high contrast to the human eye but is not discriminative to computer vision algorithms as they are operating on grayscale images Tip: For low contrast images, try to increase the contrast of your target image with an image editing tool like Gimp or PhotoShop to improve detection and tracking quality
Distribution of textured areas
- Images with evenly distributed textured areas are good candidates for reliable detection and tracking
- This might be the hardest part to be in control of and often can’t be changed. Tip: Try to crop the most prominent part of your image and use only this as target image.
Images with whitespace
- Single-colored areas or smooth color transitions often found in backgrounds do not exhibit graphical information suitable for detection and tracking. Tip: Try to crop the most prominent part of your image and use only this as target image.
Vector-based graphics
- Logos and vector-based graphics usually consist of very few areas with high local contrast and textured structures and are therefore hard to detect and track.
Tip: Try to add additional elements to the graphic like your logotype or any other specific elements, which can go along with your graphic.
Images with a lot of text
- Images consisting primarily of large areas of text are hard to detect and track.
Tip: Try to have at least some graphical material and images next to your text for your target image.
Repetitive patterns
- Repetitive patterns exhibit the same graphical information information at each feature point and therefore cannot be localized reliably
- Images with slightly irregular structures can convey a similar information to the target audience while providing enough unique feature points to be detected (second image)
Tip: Try a different selection of your image including non pattern parts or use images with irregular patterns
Reference
iOS Native API Reference
Goto iOS Reference for a complete reference of all iOS Wikitude Native API objects and functions.
Cloud Recognition Manager API
Goto REST API Reference for a complete reference of all REST API calls for the Manager API.
Wikitude SDK iOS Native API Release Notes
Wikitude SDK 5
Wikitude SDK Native API 1.0.0 beta
Release Date: 30.07.2015
New
- Initial Public Release of iOS Native API