The Return to VR / AR / Games

The First Commercial VR Game – VR Slingshot in 1994

I created the first virtual reality game for the PC in 1994: VR Slingshot. It featured realistic and accurate custom physics (no libraries back then), custom generalized fluid dynamics for airflow simulation (back then flight simulators used limited lookup tables), 2-person lag-free multiplayer via modem, stereoscopic 3D rendering, 3D audio, and in 1995 supported the first head-mounted displays and head trackers, including the Virtual i-O I-Glasses which I helped develop.

In 2001 I helped develop the first XBox Live! game, Wacked!, and in 2004 released Amazing Curves Racing, using updated custom physics and fluid dynamics, accurate dynamic tire simulation directly on cubic curved surfaces (instead of polygons), dynamically tessellated cubic 3D surfaces for the track (3D cubic Bezier surface model data exported via custom plugin for 3DSMax), and up to 64-player multi-player via the internet, where lag was well hidden even with pings as high as 500ms.

In 2006 I moved to Beverly Hills, CA to work for MySpace and wrote audio/music and video software. Since then, I’ve worked on many interesting projects, though no games or VR. After taking a break from tech and working on very creative projects with cameras and video, I saw the latest in AR tech for the iPhone and wondered how well a phone could run some of my algorithms for physics and gaming in 2018.

Using an iPhone X I wrote test code with a further updated fluid dynamics model in Swift, a language I hadn’t used before. I write all high performance math code in C++, and was curious if Swift would be fast enough. Another reason to use Swift on the iPhone is it’s much easier and faster to develop with all Apple-related code elements vs. Objective-C, which I’ve used before (and never was a fan).

Starting with an AR test app, I got 400 fluid dynamics simulated jets flying around at 60+fps on the iPhone X in Swift. This is using the relatively high polygon count default art “Jet” from Apple. The last time I worked on game code, it was necessary to use progressively lower polygon levels of detail objects as GPUs weren’t nearly as powerful as they are today. The iPhone X is significantly more powerful than the PCs+GPUs I last wrote games for! Pretty impressive.

400 Jets @ 60fps – iPhone X

I noticed there was a lot of interest for VR+AR for phones, and found simple VR headsets for phones with decent optics for around $35 or less. I ordered one of these EXCLEAD headsets for $34.99.  I chose this unit for development as it’s easy to get the phone in/out, would also be cooler to keep the phone from heat-throttling, and also left the camera uncovered so it would be possible to to mixed AR/VR. Additionally this unit provides inter-pupillary distance and diopter adjustments (not present on many other units). After receiving the unit, phone fit well and the optics were decent, the only issue is an overhead strap is needed to help with weight distribution. I fashioned an overhead strap from elastic material and velcro I had on hand from camera lighting equipment.

I tried some mix AR+VR demos using the iPhone X camera to provide tracking, and found that it jumped around a lot and the tracking was a bit laggy compared to Google Cardboard-based VR-only demos/tests. I then moved forward focused on VR only using code which included an Extended Kalman filter which does a decent job of predictive filtering to help reduce apparent lag.

I tested Unreal Engine and Unity 3D and found performance to be pretty good. However, when working with art assets and importing / building scenes, these tools were very, very slow compared to just using Swift in XCode. For more advanced games, the tables will turn as managing/building art assets becomes more complex. For simple physics-based VR experiences, Swift+XCode as fine. During initial development I added support for calling code written in Objective-C and C++; thus advanced physics can run in C++ if necessary.

The final tech element to develop was rendering both eyes efficiently. Apple provides all the pieces, you just need to figure out which elements to use. I use SceneKit to manage the 3D scene, then custom Metal shaders to render the warped image for the VR HMD optics. Now that the core technology is running, what is the game?

VR Slingshot 2.0

VR Slingshot for the PC was actually a sport, or cyber sport, which combined realistic flight simulation with a soccer-like sport (+ two simpler games based on energy). You pick up a ball and carry it on your jet, then shoot the ball through your opponent’s goal. Your opponent could shoot the ball off your plane and pick it up and vice-versa. This is the perfect simple, fun kind of game for a return to game development and VR. VR Slingshot 2.0 (working title: name may change) will be the same basic game with updated graphics, and might use some of the original sounds for a retro experience. The first version will be single player (against AI opponents), and if multiplayer is added, more than two players will be able to play (perhaps up to 8: 4 vs. 4).