Over at our time at the Games Developer Conference (GDC 2023), we had the opportunity speak with Zhang Jian, the head of PICO XR Engine and Runtime Engineering in an interview.
Zhang Jian has 10 years of experience in the console game industry and 7 years of VR industry experience. In his career in the games industry, he has shipped games like Call of Duty, Ghost Recon, Tom Clancy’s EndWar. During his time in the VR industry, he has worked on Oculus Rift, GearVR, and Oculus Quest.
Prior to his 9 months in PICO, Zhang Jian was the Tech Lead at Meta. His area of expertise includes XR Graphics, XR Runtime, and Game Engines.
Interview is edited for clarity
Considering each component of XR, AR, VR and other specialised technology, how does PICO prioritise which field to give more focus to?
Zhang Jian: PICO’s headsets are primarily focusing on VR currently. However, you can see that the industry is trending towards to Mixed Reality.
Our latest headset is PICO 4, which has some level of MR capability with video see-through. If you haven’t already, do try the PICO 4 and check out the color see-through feature.
You used to work for AAA game companies. What is the difference like between working in that environment as compared with PICO?
Zhang Jian: It’s quite different. For AAA games they are somewhat more like movies, which take a long time to finish, where a lot of AAA games developments take more than three years. For example, Call of Duty had three studios and each game took three years, rotating between different development teams. You have this very clear goal that you’re going to finish this at a certain time. However, the tech company like PICO is much more fast paced, technology is rapidly iterating.
We used to work on one product but right now as a platform, we serve a lot of developers. So that’s the difference in the role. We have developers make the game for our platform instead of us making the game directly.
Now that you’re working with a lot of different developers, what kind of support do you give them?
Zhang Jian: Here in GDC, we’ll talk about the developer side of things. For us, we have game engine support, software development kits, and developer tools.
With other companies, they have test kits and debug kits that they would send to developers. What kind of tools does PICO send to developers?
Zhang Jian: That’s the very interesting part. I worked for the console industry before, and you actually buy really expensive development kits to be a developer. Also not every company can buy them even if they want.
With PICO, basically every consumer has the ability to create their own development kit very easily. We have some instructions on our website that show you how to do that. It only needs a few steps, and you can enable your headset’s developer mode. This essentially means that every consumer can potentially become a developer.
I think the biggest challenge is always how some people can feel dizzy and get motion sickness when using VR. From an engineering perspective, how do you combat this?
Zhang Jian: There are two sorts of motion sickness. I think one part is coming from the basic metric. Currently in VR, you have to guarantee your experience runs very smoothly and can hit a frame rate of around 72 or 90 FPS. That’s the very hard part, you have to be there at that frame rate and if you keep losing frames, you’ll get sick.
So we will develop our tool, optimize our tech stack to improve the performance. We also work with the developers and give them the tools to solve this kind of issue. And in our Core tech stack, there is another metric we call it latency, which can affect VR comfort a lot, it is quite important to keep optimizing latency.
You mentioned that FPS Freeze or Low FPS often create motion sickness. I do play games that have low FPS but still do fine. But certain games with high FPS, I still feel dizzy. I don’t think it’s just the frame rate problem, perhaps there could be other problems that cause motion sickness?
Zhang Jian: Another problem might be the optical side of things like distortion. That’s one of the metrics on the hardware side. Thinking from the develop side, what we can control is more like the FPS and the latency. And on that part we develop our tools to help people do that. I think you mentioned a good point. When you play a flat screen FPS game, the frame rate being around 20 and you don’t get sick.
I think the environment in reality is super stable in your vision. For example, when you are moving your head, the table in front of you are staying at the same place quite stably; but in VR, if your FPS is low, you will see the environment is actually not completely fixed in term of spatial position.
That’s the part that makes you sick. Also when your latency is high, your environment is even less stable.
But when I move, it doesn’t move right?
Zhang Jian: Yeah. It’s not what normal people can see, and that’s from a technical point of view. There is another stuff that affects your motion sickness from a design point of view. Like one example would be if you use your controller to move the camera smoothly.
That’s super uncomfortable because your brain tells you “I’m not moving” but somehow you’re really moving from vision point of view. This kind of mismatch will cause people to want to throw up.
When designing games, you normally don’t need to care about the space of the play area in real life. But for VR games, you’ll need to. Do you put that into consideration to help developers when they design their games? Do you think it would actually affect the development side of games?
Zhang Jian: I think this is more of a developer decision and what kind of game they would want to make. If they make a game that needs a large space but not everyone has that kind of space, some consumers are not going to buy the game or not be able to play it very well. They need to decide who are their game’s primary target.
Let’s say this game developer already has existing games on Android and iOS, and they want to deploy those same games to the PICO experience. Do they need to build something totally new or can they port the game quickly to people?
Zhang Jian: PICO is based on Android OS, right? The application is an APK, exactly the same with other Android games. If you have the game project on for example in Unity or Unreal Engine, it’s actually pretty easy to enable PICO integration. You just need to fill some checkboxes, import some stuff and then it’ll work.
So with Unity and Unreal, there’s not much extra time needed to port?
Zhang Jian: Yeah, not much time to make it work on PICO. But it does need a lot of effort to make a good experience in VR, because if you design it for the phone, you probably didn’t consider too much about the VR experience – like earlier when we mentioned motion sickness.
For example, you probably wouldn’t want to hijack the VR camera and do camera animation, that’s not good for VR comfort.
If I was a developer and I developed a 2D side-scroller platformer game and wanted it on PICO, do I need more time to put it into your platform or is it as fast as in a few months?
Zhang Jian: The integration stuff will be fast. However, the polishing and optimisation will take some time.
University students who only learn about Unity and Unreal Engine, is that sufficient enough for them to join your department?
Zhang Jian: That depends on the opening positions. We have different positions for different roles. Some positions will require a certain level of computer graphics knowledge and that’s a little bit hard for some people.
For application development like what you mentioned, e.g. how do you enable Unity app running on VR, a lot of people can do that. However, we do hire new graduates from university.
VR now is all about eyes and hands to control the experience of things. What’s the next step for VR to increase the immersive experience and do you have more future plans on this?
Zhan Jian: Yeah. There’s definitely a lot of stuff there but I’m afraid I can’t talk more in detail. But one thing is, we got the hand tracking to work. Right now it’s in the early stages and we’ll continue to work on that to improve it.
Any past experience you had that benefited you now that you’re in PICO?
Zhang Jian: I think the industry so far is pretty open. We have an open standard called Open XR, and a lot of VR manufacturers and developers contribute there. We talk with each other and when one developer has a good idea, they would share a blog post or tech notes so that everybody can learn. So that’s how the industry is helping each other.
What are your hopes for the VR industry?
Zhang Jian: There is definitely hope that the whole industry goes forward and becomes bigger, and has more people wanting to jump into this space to play games and develop games here. Generally, we are not only hoping for ourselves to succeed, we also hope everyone succeeds as well.
We would like to thank Zhang Jian for taking the time to answer our questions. We would also like to thank PICO for helping us arrange the interview. For more information on PICO products, do check out their official website here.
Discussion about this post