Augmenting Augmented Reality

Blog Post created by dourada on Aug 26, 2016

The HoloLens has been an enjoyable addition to the Center for Innovative Payments in our Durango facility.  We always enjoy new technology and plotting how the technology will fit into our world and the world of our partners.  After a few glimpses of static holograms and of course the stunning Galaxy Explorer I wanted to learn more about the actual coding.  How exactly are people making holograms?  And what kind of effort would it take to augment a holographic application with the ability to pay?  Two great questions that will be answered in this post.


First we will discuss "programming" a holographic component.  There are many tutorials but the most enjoyable is taking a look at the actual code used to build the Galaxy Explorer.  Luckily the team open sourced the project so building and deploying to the HoloLens Emulator was straightforward.  Before you start installing make sure to check the prerequisites in the link below.



This deployment utilizes Unity as the development environment and then Unity builds a Visual Studio Solution that you can use to build/run the actual code in the HoloLens Emulator.


However, I wanted something a bit easier as a 'tracer' through the system.  Is it possible to build a holographic component directly from Visual Studio?  The answer is....of course.  If you already installed the tools above your environment is ready to go.  If you did not already install follow the steps in the 'Install the tools' link above.


After you have installed the HoloLens tools open Visual Studio and select File-->New-->Project.  You should see a screen like below.  I am going to create a Holographic c# application from the preinstalled template.  Give your solution a name and press the ok button.



The amazing thing is...that's the end.  You have now created an app to deploy to the HoloLens Emulator.  Here's what the solution looks like after creation and notice in the Target windows that HoloLens Emulator has been selected.



Press the green arrow or hit F5 and let's see what happens.  Nuget package manager will restore any packages needed, the solution will compile, and then after about a minute the project will be deployed to the HoloLens Emulator.  It takes a while for initial setup but if you're patient you will eventually see a revolving colored cube.



You can experiment with all of the controls on the right hand side but the basics are use the mouse left click and hold to move the camera around as if you are moving your head with the HoloLens on.  Or you can use the "w-a-s-d" keys to move the camera around the scene.


Now that we have the basics how do we add payment functionality?  Pretty easy.  Let's assume that there is a two step payment process and the first has already been completed.  The first step is signing up online to receive/install the spinning cube HoloLens application.  You used one of our e-commerce applications like Hosted Checkout right to accept payment from spinning cube customers.  After the cardholder submitted their card data via the Hosted Checkout application you as the app developer received a token in the response and saved it for future use.  We can use this token to make payment requests inside of our HoloLens application.  So let's see how we can click (or Air Tap) on the rotating cube to generate a payment event.


Amazingly there is already code stubbed out that helps us out.  In the Content folder look for SpatialInputHandler.cs and open that file.  Then look for the OnSourcePressed event.  Your dev environment should look something like the screenshot below.



If you are already familiar with c# and sending a transaction to the MercuryPay platform it's as easy as copy/pasting the code in here, compiling, and executing.


I added the following lines of code to the SpatialInputHandler.cs file after the OnSourcePressed method: 


bool MakePayment()
                var requestToSend = @"{
                                    ""Memo"":""Go HoloLens"",


                return true;

            return false;

        async void DoRestRequest(string json)
            var dataByte = UTF8Encoding.ASCII.GetBytes("337234005:xyz");
            var merchantIdData = Convert.ToBase64String(dataByte);

            using (var client = new HttpClient())
                client.BaseAddress = new Uri("");
                client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
                client.DefaultRequestHeaders.Add("Authorization", "Basic " + merchantIdData);
                var stringContent = new StringContent(json);

                var response = await client.PostAsync("PaymentsAPI/Credit/Sale", stringContent);


Finally I added a call to MakePayment() in the OnSourcePressed method after the sourceState = args.State.


There was one tricky bit that I did not expect.  Because we're building a windows 10 Universal application we have to specify in the Package.appmanifest that the application is using the Internet.  Otherwise you'll receive an ugly error message, "a network capability is required to access this network resource," when running the application.  Double click on Package.appmanifest in solution explorer.  Find the Capabilities box and check Internet (Client).



Place a breakpoint on the MakePayment() call in OnSourcePressed event.  Build and run the application in HoloLens Emulator.  When the spinning cube is displayed place the mouse cursor over the cube and press the right mouse button.  This action is synonymous to moving your head around with the HoloLens on and aiming your gaze at the spinning cube and then Air Tapping or pinching your thumb and index finger together over the spinning cube.   This will fire the OnSourcePressed event and you will be able to step through the code and see the payment transaction complete.  Logging into our certification portal shows the following.  Note the highlighted row at the bottom with the operator = HoloLens!


This is obviously the "Hello World" simple/quick version of how to integrate payments into a HoloLens application.  There is so much more to learn and do so keep checking back for further deep dives into augmenting HoloLens with payment functionality.


We would love to hear from you if you are working on HoloLens applications or if you would like to specifically discuss how to add payments to your virtual reality or augmented reality applications.


Happy Coding!