WorkA selection of previous projects.
Ultrasound Training Application
A multiuser extended reality ultrasound training application for medical students to practice point-of-care ultrasound in a simulated environment.
Collaborators
Leiden University Medical Center (LUMC)Leiden Learning and Innovation Centre (LLInC)Tessa MulderManon Zuurmond
Role
Unity Developer
Objective
Create a multiuser HoloLens II application that allows medical students to practice obtaining accurate cardiac ultrasound images.
Challenges
• The HoloLens II does not have a controller which is needed to act as the ultrasound probe.• The multiple HoloLens devices, the phones, and the 3D printed patient body had to be spatially calibrated to the same environment.
Solution
• A custom built networking solution using UDP / WebSockets was built to allow the HoloLens devices and the phones to communicate with each other.• The phone was used as a virtual ultrasound probe and uses AR to send its pose, and contained the main user interface for the application.• Image tracking with Vuforia for the HoloLens, ARKit for iOS, and ARCore for Android was used to spatially calibrate all devices in the same space. - The animated mesh of the heart is clipped at runtime with a custom shader written to generate the ultrasound image.
Outcomes
An application that allows multiple users to join the same training environment, practice the correct imaging angles with virtual indicators placed on the 3D printed patient body, and evaluate their performance. The application was built in two versions.• The 1st version was built for the HoloLens II with a companion app for iOS and Android, the app acts as the virtual probe and main control user interface.• The 2nd version was built for Meta Quest 3, and Logitech MX Ink stylus was used as the ultrasound probe, this version improved accuracy by a high margin as it relied on natively supported tracking instead of a custom networked solution.
Publications
The applications allowed recording the users performance, and was evaluated, with the results published in two peer-reviewed journals.
Interactive Media and Virtual Production Studio
A studio that integrates filming equipment, lighting, projectors, trackers to create an interactive simulated environment for learning, education, and media production.
Collaborators
Leiden Learning and Innovation Centre (LLInC)
Role
Lead Designer and Developer
Objective
Create a space that allows LLInC to experiment with latest media technologies in their innovation work, and create a virtual production studio.
Challenges
• The studio needed integration of a large number of incompatible hardware, software and protocols.• The studio needed to be operated by a friendly user interface.
Solution
Create a custom software built using Unity to act as the main control and modelling environment. The application allows users to import media, create the scene, control lighting, cameras, microphones, trackers, projectors, and record the results.
Outcomes
The studio became an innovation space for LLInC where they can experiment with different technologies under one roof, operated by one custom built software.
Augmented Reality (AR) Assembly Manual
A mobile AR application that guides users into assembling flat-pack furniture in a gamified experience as part the thesis for my M.Sc. in Software Development.
Context
University of Glasgow
Role
Unity Developer
Objective
Create a mobile AR assembly manual for a flat-pack chair that tracks the pieces and provides augmented instructions to the user..
Challenges
Each piece of the chair had to be tracked by QR codes attached to it, however, the images could be hidden from the view.
Solution
• Add multiple images to each part and calibrate them all to the same origin.• Vuforia was used for tracking as it could handle a large number of images which was not available by ARKit at the time.
Outcomes
The application was built for iOS and was tested against the paper manual in a controlled group experiment with the results written as part of the overall M.Sc. thesis.
Clowny Rain
A mobile iOS arcade game.
Context
Personal Project
Role
Unity Developer
Description
A fast shooting arcade game developed using Unity 3D and Entitas, an Entity-Component-System framework. The game was published on Apple App Store.
Augmented Reality (AR) Flooring Visualiser
A mobile AR application that allows users to visualise different flooring solutions in their house or property.
Client
Upwork Client
Role
Unity Developer
Goal
Create an AR flooring visualisation app that allows users to view different floorings from the retailers database.
Challenges
• Occlusion and tracking using ARKit and ARCore are not accurate enough for all cases.• The Unity app had to be embedded in a larger native application. The different floorings has to be retrieved from the client’s database.• Create realistic rendering.
Solution
• Create a manual floorplan mapping solution with the capability to create geometric shapes that gets subtracted from the rendering for furniture.• Generate PBR materials at runtime from the textures stored on the database.• Use AR Light Estimation and Depth occlusion for realistic rendering.
Outcomes
A Unity application using ARFoundation (ARKit for iOS and ARCore for Android) was developed and embedded in the larger mobile application with flooring visualisation, floor textures retrieval, and rendering.
Augmented Reality (AR) Assistant
A Virtual Assistant with Text-to-Speech and Speech-to-Text embedded in the user’s environment.
Client
Upwork Client
Role
Unity Developer
Goal
Create a friendly robot butler that acts as an AI assistant embedded in the user’s environment.
Solution
• Use Watson Text-to-Speech, Watson Assistant, and Text-to-Speech to allow the assistant to interact with the user.• User ARKit to embed the assistant in the user’s environment.• Integrate AIML files with Watson to allow full conversation with question answering, jokes, and play games.
Outcomes
An AR iOS application developed using Unity and Watson Assistant utilities.