29 June 2016

Gotcha - changing a HoloLens app’s name as it appears on the tile

I am currently finishing up a nice little demo app for the HoloLens but ran into a weird problem. Originally the app was called ‘Gazer’, as I only wanted to test Gaze Input, but it turned out to become a lot more. So I wanted to call it “CubeBouncer”.

It’s not so hard to do, you got File/Build settings, then go to “Player Settings”, and in the inspector on the right hand side you see a Store logo. Click that, select the Icon tab, and under “Short name” you will see the name that will be displayed on the HoloLens app tile. You can also select on what types of tiles it needs to appear. It quite reflects the Visual Studio UWP Manifest editor, which is logical, as this is what the manifest is generated from.

image

Build the app by hitting the Build button as usual when you have made changes in Unity, Visual Studio will say the project has changed and prompts to reload it, you build and deploy it to the HoloLens or the emulator, pin your app to the start screen – and you will see nothing has changed. The old name is still displayed.

Turns out that Unity UWP generation is a bit too clever when it comes to generating the UWP – apparently it updates not everything, but only the things that have changed. And I think it fails to take changes into the manifest into account.

The solution is very simple: don’t overwrite the generated UWP app. Close Visual Studio, delete the generated UWP app entirely (warning – only the generated app code, not your entire Unity project), only then hit Build in Unity (you will notice that takes quite a bit longer, as it needs to restore NuGet packages and stuff too), open Visual Studio again and then if you deploy your app to HoloLens, the short name will have changed.

Possibly there are smarter ways to do this, like only deleting the Manifest file – I have not tried this – but this is a sure fire way to fix this. It’s a bit cumbersome, but changing the name of an app or a tile isn’t exactly something you do ten times a day, so it’s not that much of a problem. But it’s a nice gotcha, so I thought it best to document it.

11 June 2016

Floating ‘text balloons’ for context relevant information in Xamarin Forms

Picking the challenge apart

An Italian proverb says that a fool can ask more questions than seven wise men can answer – a modern variant could be that a designer can think up more things than a developer can build. I don’t pretend to be the proverbial wise man, and neither do I want to call my designer colleague a fool, and when he came up with the idea of introducing text balloons with context relevant information, floating on top on the rest of the UI, cross-platform, and preferably appearing with a nice animation, I indeed had to do some head scratching.

What he meant was this, and this is exactly how I created it

The issues I had to tackle, were:

  1. How do I create a text balloon in the first place, with a kind of pointy bit pointing to the UI element it belongs to?
  2. How do I get the absolute position of a UI element - that is, the one the user taps?
  3. How do I show the text balloon in situ?

Text balloon 101

tekstballon

This text balloon consists of a translucent grid that ties the components together. It contains two grids, one of them containing the label. The first grid is the bit that points up. This actually is a 15 by 15 square, rotated 45⁰, and moved a little bit to the left using the new Margins property that has finally made it to Xamarin Forms. The second and biggest grid is the green rectangle actually containing the text you want to show. Because it’s in XAML after the square, drawing precedence rules make that it be drawn on top of the first one. You would not see it at all, if is wasn’t for the fact this grid also has a margin - of 7 on the top so about half of the rotated square. The net result, as you can see, is a triangle sticking out of the rectangle, making the optical illusion of a kind of text balloon. In XAML, this looks like this

<Grid x:Name="MessageGridContainer" xmlns="http://xamarin.com/schemas/2014/forms"
           xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
           x:Class="XamarinFormsDemos.Views.Controls.FloatingPopupControl" 
           BackgroundColor="#01000000">
  <Grid.ColumnDefinitions>
    <ColumnDefinition Width="75*"></ColumnDefinition>
    <ColumnDefinition Width="25*"></ColumnDefinition>
  </Grid.ColumnDefinitions>
  <Grid x:Name="MessageGrid" HorizontalOptions="Start" VerticalOptions="Start" >
    <Grid BackgroundColor="{StaticResource  AccentColor}" HeightRequest="15" WidthRequest="15" 
          HorizontalOptions="End" VerticalOptions="Start" Rotation="45" 
          Margin="0,0,4,0" InputTransparent="True"/>
    <Grid Padding="10,10,10,10" BackgroundColor="{StaticResource AccentColor}" Margin="0,7,0,0" 
          HorizontalOptions="FillAndExpand" InputTransparent="True">
      <Label  x:Name="InfoText" TextColor="{StaticResource ContrastColor}" 
             HorizontalOptions="Center" VerticalOptions="Center" 
             InputTransparent="True"/> 
    </Grid>
  </Grid>
</Grid>

The complete text balloon is contained in the “MessageGrid” grid; the pointy bit upward is emphasized using red and underlining. The complete control is contained within yet another grid “MessageGridContainer”, that fills the whole screen – or at least the part in which the text balloons appear. Is has three functions:

  • It provides a canvas to place the actual text balloons on
  • It is an event catcher – as the user taps ‘anywhere on the screen’ the text balloon disappears
  • It makes sure the text balloon never gets wider than 75% of the screen (this was a designer requirement) – hence the columns.

Some important details to take note of:

  • A few elements have set the property “InputTransparent”  to true. This means they will never receive any events (like tap) but those will be received by the elements lying ‘below’ them. In other words, they don’t block events. This makes the text balloon disappear even when you tap on the text balloon itself, as the events goes downwards and is processed by MessageGridContainer
  • MessageGridContainer itself is not opaque but has BackgroundColor "#01000000", that is, 1% black. For all intents and purposes it is opaque in the sense that you don’t see it, but if you leave it totally opaque is will also be opaque to events - on Windows 10 UWP. A little concession to a cross platform issue.
  • This whole contraption is called “FloatingPopupControl” – this is the control that handles showing, displaying and eventually removing the text balloon. ‘Something’ has to call it’s ShowMessageFor method to tell it what the balloon should contain, and under which control it should appear.We will come to that later

Determining absolute position of the ‘anchor element’

The anchor-element is the element under which the text balloon appear should appear when it’s tapped – in this sample, the i-symbol. It is actually pretty to simple to find the absolute position of a relatively placed element: this is achieved by going recursively upwards via the “Parent” propertie and get the sum of all X values and the sum of all Y values. You can actually find hints to this in the Xamarin developer forumsin the Xamarin developer forums, and I have put this into the following extension method:

using Xamarin.Forms;

namespace Wortell.XamarinForms.Extensions
{
  public static class ElementExtensions
  {
    public static Point GetAbsoluteLocation(this VisualElement e)
    {
      var result = new Point();
      var parent = e.Parent;
      while (parent != null)
      {
        var view = parent as VisualElement;
        if (view != null)
        {
          result.X += view.X;
          result.Y += view.Y;
        }
        parent = parent.Parent;
      }
      return result;
    }
  }
}

Positioning, showing, animating and removing text balloons

If you look at the code in the ShowMessageFor method - in the FloatingPopupControl code behind – you’ll see the code is only deferring to FloatingPopupDisplayStrategy. This is done because it’s not wise to put much code into a user control if you want to re-use part of that intelligence easily. It also makes adapting and changing animations easier. FloatingPopupDisplayStrategy has the following constructor:

public class FloatingPopupDisplayStrategy
{
  private readonly Label _infoText;
  private readonly View _overallView;
  private readonly View _messageView;

  public FloatingPopupDisplayStrategy(Label infoText, View overallView, View messageView)
  {
    _infoText = infoText;
    _overallView = overallView;
    _messageView = messageView;

    _overallView.GestureRecognizers.Add(new TapGestureRecognizer
    { Command = new Command(ResetControl) });
    _overallView.SizeChanged += (sender, args) => { ResetControl(); };
  }
}
  • infoText the text balloon name 
  • overallView is the canvas in which the text ballon is placed; it also receives the tap to remove the text balloon again
  • messageView is het containing grid of the text balloon itself

ShowMessageFor, and it’s little helper ExecuteAnimation, are implemented like this:

public virtual async Task ShowMessageFor(
  VisualElement parentElement, string text, Point? delta = null)
{
  _infoText.Text = text;
  _overallView.IsVisible = true;

  // IOS apparently needs to have some time to layout the grid first
  // Windows needs the size of the message to update first
  if (Device.OS == TargetPlatform.iOS || 
      Device.OS == TargetPlatform.Windows) await Task.Delay(25);
  _messageView.Scale = 0;

  var gridLocation = _messageView.GetAbsoluteLocation();
  var parentLocation = parentElement.GetAbsoluteLocation();

  _messageView.TranslationX = parentLocation.X - gridLocation.X -
                              _messageView.Width + parentElement.Width +
                              delta?.X ?? 0;
  _messageView.TranslationY = parentLocation.Y - gridLocation.Y +
                              parentElement.Height + delta?.Y ?? 0;

  _messageView.Opacity = 1;
  ExecuteAnimation(0, 1, 250);
}

private void ExecuteAnimation(double start, double end, uint runningTime)
{
  var animation = new Animation(
    d => _messageView.Scale = d, start, end, Easing.SpringOut);

  animation.Commit(_messageView, "Unfold", length: runningTime);
}
  • First the text that should be displayed in the text balloon is set
  • Next, the canvas in which the text balloon is placed is made visible. As stated before, it’s nearly invisible, but effectively it intercepts a tap
  • Windows and iOS now need a short timeout for some layout events. This feels a bit VB6’ey, doevents, right?
  • The text balloon is scaled to 0, effectively making it infinitely small (and invisible) 
  • Next, we calculate the text balloon’s current absolute location, as well as the anchor element’s(‘parentElement’) absolute location.
  • X and Y translation of the text balloon are calculated to position the text balloon at a location that will make the pointy bit end up just under the blue i-symbol
  • De message grid’s opacity is set to 1, so now the text balloon is visible (but still infinitely small)
  • A 250 ms bouncy animation (Easing.SpringOut) blows up the text balloon to scale 1 – it’s normal size.

Note: the delta uses in the calculation is a value intended to use as a correction value, in case the standard calculation does not yield the desired result (i.e. location). This will be explained later on.

And finally, the user must be able to dismiss the text balloon. This is done using the ResetControl methods. As we have seen in de constructor, this method gets called in case the user types at the invisible canvas, or if the canvas’ size changes.

private void ResetControl()
{
    if (_messageView.Opacity != 0)
    {
      _messageView.Opacity = 0;
      _overallView.IsVisible = false;
    }
}

This method does not need to be called explicitly at initialization, since he invisible grid changes size at the start of the app (because it gets child elements – the MessageGrid and its children), and the event wiring makes this call happen anyway. Another important reason to attach this method to the SizeChanged event is that in Windows 10 UWP apps windows sizes actually can be changed by the user. This may cause text balloons ending up in what is no longer being the right place, so they need to be removed as well. After all, as long as the text balloon is visible, the invisible background blocks any input, so as soon as the user starts working with the app, in any way, the text balloon needs to disappear and the app needs to be ready again.

Behavior intercepting tap event and relay to control

The only thing missing now is something to get the whole process going – respond to the tap on the i-symbol, providing the text balloon contents, and provide some optional positioning correcting for the text balloon. This is done by FloatingPopupBehavior:

using Wortell.XamarinForms.Behaviors.Base;
using Wortell.XamarinForms.Controls;
using Xamarin.Forms;

namespace Wortell.XamarinForms.Behaviors
{
  public class FloatingPopupBehavior : BindableBehaviorBase<View>
  {
    private IGestureRecognizer _gestureRecognizer;

    protected override void OnAttachedTo(View bindable)
    {
      base.OnAttachedTo(bindable);
      _gestureRecognizer = new TapGestureRecognizer {Command = new Command(ShowControl)};
      AssociatedObject.GestureRecognizers.Add(_gestureRecognizer);
    }

    protected override void OnDetachingFrom(View bindable)
    {
      base.OnDetachingFrom(bindable);
      AssociatedObject.GestureRecognizers.Remove(_gestureRecognizer);
    }

    private void ShowControl()
    {
      if (AssociatedObject.IsVisible && AssociatedObject.Opacity > 0.01)
      {
        PopupControl?.ShowMessageFor(AssociatedObject, MessageText, new Point(Dx, Dy));
      }
    }

    #region PopupControl Attached Dependency Property      
    public static readonly BindableProperty PopupControlProperty =
      BindableProperty.Create(nameof(PopupControl), 
      typeof (IFloatingPopup), typeof (FloatingPopupBehavior),
        default(IFloatingPopup));


    public IFloatingPopup PopupControl
    {
      get { return (IFloatingPopup) GetValue(PopupControlProperty); }
      set { SetValue(PopupControlProperty, value); }
    }
    #endregion

    //MessageText Attached Dependency Property omitted

    //region Dx Attached Dependency Property omitted     

    //region Dy Attached Dependency Property omitted     

  }
}

This behavior is actually rather simple – as soon as the control tot which is attached is tapped, it calls the ShowMessageFor method of the control referenced in de PopupControl property. There are three additional property for determining which text is actually displayed, and two optional properties for a delta X and delta Y which, as we have seen, are included by the control when it actually places the text balloon on the right place.

Bringing it together in XAML

A simplified excerpt from FloatingPopupPage :

<ScrollView Grid.Row="1"  VerticalOptions="Fill" 
   HorizontalOptions="Fill" Margin="10,0,10,0" >
  <Grid>
    <Grid VerticalOptions="Start">
      <Grid.RowDefinitions>
        <RowDefinition Height="Auto"></RowDefinition>
        <RowDefinition Height="Auto"></RowDefinition>
        <RowDefinition Height="*"></RowDefinition>
      </Grid.RowDefinitions>
      <StackLayout Orientation="Horizontal" HorizontalOptions="Fill" >
        <ContentView  HorizontalOptions="FillAndExpand" VerticalOptions="Start">
          <Entry x:Name="NameEntry" Placeholder="Name" 
                  TextColor="{StaticResource AccentColor}" 
                  PlaceholderColor="{StaticResource SoftAccentColor}" />
        </ContentView>
        <ContentView>
          <Image Source="{extensions:ImageResource info.png}" VerticalOptions="Center"
                 HorizontalOptions="End" 
                 HeightRequest="{Binding Height, Source={x:Reference NameEntry}}">
            <Image.Behaviors>
              <behaviors:FloatingPopupBehavior MessageText="Fill in your name here"
                                               PopupControl="{x:Reference PopupControl}" 
                                               Dx="-6" Dy="4"/>
            </Image.Behaviors>
          </Image>
        </ContentView>
      </StackLayout>

    </Grid>
    <controls:FloatingPopupControl x:Name="PopupControl" VerticalOptions="Fill" 
                                   HorizontalOptions="Fill" />
  </Grid>
</ScrollView>

In red the actual i-symbol with the behavior attached to it, in green the popup control (including the label) itself. In the behavior’s properties we actually specify the text to be displayed as well the PopupControl reference, indicating this is the UI control that should actually handle the displaying of the text balloon. In addition it sports an optional extra delta x and delta y. Of course this could be hard coded into the control, but to have this extra flexibility in design time makes for an easier ‘constructable ’UI. As you can see, as soon as the parts are in place, actually using and re-using the components is pretty easy, making adding floating text balloons with contextual relevant information very easy indeed.

Also notice a neat trick to make sure that especially nice in Android, that sports a great rage of resolutions. I took an intentionally too big picture for the i-symbol, which is automatically sized to the height of the entry by using  HeightRequest="{Binding Height, Source={x:Reference NameEntry}}"

Some consequences of this approach

As stated repeatedly, a (nearly) invisible grid covers the whole screen, or at least part of the screen, while a text balloon is displayed – to give the text balloon space to be placed in, and to intercept a tap so it will be removed as soon as the user starts interacting with the app. The flip side is that the app is effectively blocked until the user taps, and this tap will not do anything but removing the text balloon. A plainly visible button will not respond while the text balloon is visible – that requires yet another tap. This may seem annoying, but I don’t think this will put off the user in any significant amount, as it;s likely he will stop using this functionality pretty soon as he/she has gotten the hang of the app. This is only an onboarding/adopting thing. You read the car’s manual only once (if you do it at all, and then never again unless in very extraordinary circustances)

Conclusion

Using only some pretty basic means, a few nifty tricks and a clear architectural approach it appears to be pretty simple to build a kind of re-usable infrastructure enabling the fast and flexible addition of context relevant information, which is displayed in a visually attractive way. It’s very easy to add text balloons this way, and it’s a useful tool to make onboarding and adoption of an app easier.

As usual, a sample project containing this (and previous code) can be found on GitHub

01 June 2016

Default HoloLens Toolkit occlusion script turns magenta

After following some on-line video’s and snooping through some other’s people code I felt I had to start bottom-up to get at least a feeling for how you setup a HoloLens app – even though I don’t have a HoloLens. Call it my natural curiosity. So I followed my fellow MVP Morten Nielsen’s blog – highly recommended – as he starts with the real basics. Very good if you hardly have a clue what you are doing ;)

In his third post, I ran into a snag. According to his second post, I had unzipped the HoloLens toolkit into the assets folder as instructed, and started to add occlusion to my project. Morten explains in detail what this is – it makes holograms disappear behind physical objects like a wall when they are ‘in front’ of the holograms (which, in reality, they never can be as they are projected on a screen not 2 inch from your eyes, but that is the magic of HoloLens).

So I added a rectangular box as a Hologram, having it stick out of a wall like halfway, so I was supposed only to see the front part. That I did, but I did also see something else:

image

Yikes – that is my box all right, but where did that horrible magenta color come from? To this moment, I still don’t know, but I found out how to fix it.

First of all, it really helps if, in addition to making the settings to your project that Morten describes in his first post, you tick at least “Unity C# project” as well in Build Settings:

image

I do the “Development Build” as well, although I don’t really know if this is neccesary).

This makes all the scripts end up in your Visual Studio solution as well, and what is more, you can debug them. Thus I learned that the script “SpatialMappingRenderer.cs” (in folder Assets\HoloToolkit\SpatialMapping\Scripts\SpatialMappingComponent) looks for materials “Occlusion” and “WireFrame” that are clearly not there.

image

If you debug this script (you can now thanks to the setting above) you will see both OcclusionMaterial and RenderingMaterial end up as null. The materials are in HoloToolkit/SpatialMapping/Materials but changing “HoloToolkit” to “HoloToolkit/SpatialMapping/Materials” does not have any effect.

So I went back to the Unity editor, selected the Spatial Mapping Renderer script again, changed the dropdown “Render mode” from “Occlusion” to “Material”, and that made the property “Render Material” pop up. I found the Occlusion Material on top of it, and it was accepted.

image

If I now debug the code, the Occlusion material is still null, but RenderingMaterial is set, and lo and behold:

image

Now I am seeing what I expect to see – a box sticking out of a wall.

Disclaimer – I am just happily stumbling along trough Unity3D and HoloLens, not hampered by much knowledge of either. I found a problem (that I maybe caused myself) and I fixed it. I hope it helps someone. If I messed up, maybe someone can enlighten me. Unlike some other people, I am not afraid to look a n00b because in this case, that’s just what I am.

The project itself can be found in a good old-fashioned ZIP file as I don’t want to clutter up my GitHub account with every little thing I try at this stage. I am still in the stage of ‘programming by changing things and observing their effects’, so I hope you will humor me. And maybe this will help someone, who knows.