Repository files navigation

Pocket6 is a smartphone application using ARKit to turn regular iPhones into spatial input controllers for distant displays.

• 6DOF hand motion + touch inputs: Users can interact by performing 3D hand movements in 6DOF (translation + rotation) and can reliably segment and further enhance their gestures by touchscreen inputs.

• Subtle gestures: Users can perform subtle hand motions within a small control space (e.g. 16 × 9 × 16 cm). This makes the gestural interaction more comfortable since users do not need to perform large hand motions.

• Unconstrained user mobility: Users can freely and fluently change their position, orientation, or posture while using the system. An auto-calibration algorithm will adapt the control space so that the user will stay correctly mapped to the distant display application (no user-initiated calibration required).

• No specialized hardware required: No external hardware, as external trackers or cameras, are required.

Interior Designer demo, included in the project.

Another demo, currently not included in the project.

Automated control space remapping.

Distant display app

• Clone or download the repository.
• Start Unity and open thePocket6 Distant Display App project.
• Open theMainScene from theScenes folder.
• PressPlay.
• ClickStart server in the app and let it run.
• Proceed with the Pocket6 smartphone app.

Smartphone app

• Open thePocket6 Smartphone App project in another Unity instance.
• Select iOS as the target platform under File > Build Settings > Platform > iOS > clickSwitch Platform.
• Open theMainScene from theScenes folder.
• Change the resolution of theGame window by selectingPhone X/XS 2436x1125 Portrait.
• Build the iOS app under File > Build Settings > Build > Create folder > Choose a name > clickSave.
• In the created folder selectUnity-iPhone.xcodeproj and open it in Xcode.
• Connect your iPhone to your Mac and select it as the buildDevice.
• In Xcode project settings underSigning & Capabilities select your Apple Developer SigningTeam ID,Bundle Identifier, and pressPlay.
• Open the Pocket6 app on your iPhone.
• Enter the network address (IP) of the distant display Mac/PC. You can find it in the UI of the distant display app. Make sure, that both devices are in the same local network.
• ClickConnect.
• Interact by moving your device-holding hand and observe the cursor on the distant display. Point at a furniture piece and press with your finger on the touchscreen to grab it. Alternatively, you can swipe your finger left or right to rotate the furniture piece.
• UseChange view on the smartphone, as well as the distant display, to see under the hood of Pocket6.

Advanced users, feel free to test various control space sizes and aspect ratios. These settings can be adjusted in thePocket6 Logic game object in the smartphone app project.

The project was made and last tested in Unity 2021.3+, Xcode 14.3, iPhone 13 Pro and iOS 16.4.

System requirements:

• iPhone with ARKit capabilities
• Unity
• Xcode
• Mac or PC

• We useSAFullBodyIK to manipulate the humanoid rig.
• The deprecated Unity HLAPI networking was changed toMirror.

• Improve network lag by usingSmoothSync.
• Improve input precision by using the1€ filter.
• Save the network addresses using the device's internal storage orEasy Save.

This work is based on a publication"Pocket6: A 6DoF Controller Based On A Simple Smartphone Application" in the ACM SUI '18: Proceedings of the Symposium on Spatial User Interaction 2018. The publication also includes related work, user studies, applications, and future work directions.

Copyright (C) 2023 Teo Babic