Accelerating Robotic Innovation through Open Source Technologies, Part 2
Is a one-stop solution in robotics simulators achievable? In Part I of this blog series, we talked with a panel of industry experts about the importance of simulation in robotics, and dove into current developer challenges. In this blog, a panel of industry experts shares their perspectives, explores the ecosystem of simulation tools, and discusses key considerations that go into choosing the best tool for your application or use case.
Q: We’ve seen a recent explosion of different simulators emerge. Everyone seems to be searching for the holy grail of a one-stop solution. Is this search realistic? And what do you look at when you’re selecting a simulator?
Adam Dabrowski: We have quite a variety of simulators. We have a traditional simulator, which has a large community and a huge importance in the robotics community, which is Gazebo Ignition, or Fortress. And it’s important to preface that with the fact that the Robot Operating System (ROS), is the most important standard in robotics. So when we’re talking about simulation for robotics, the integration with ROS is paramount. Then we have simulators that are based on game engines. The reason for using game engines is that they provide a lot of assets, free value, performance, and so on. And we have a variety of approaches for licensing as well, including open source licenses. And the Open 3D Engine (O3DE) comes in here, which is a very promising new open source simulator.
Lars Gleim: We’re seeing an explosion of new simulators, especially around and integrated into games, but probably a lot of them will remain prototypes or applicable to specific areas. Depending on the robotics application or use case you’re targeting—whether it’s drones or service robots or marine robots—you have very different requirements in terms of feature sets, as well as trade-offs for the implementation.
What we are seeing with engines like O3DE, and other game engines with robotics integrations, is that they can really benefit from an ecosystem. The game ecosystem has developed over many years, and there are a lot of ready-made tools available to integrate with virtual reality headsets, to integrate with asset stores, to integrate with the best tools in 3D world creation, and so on. This is one of the strengths that O3DE brings for me moving forward—it’s an open ecosystem that a lot of major players are contributing to. You can ask, “Why O3DE? There are other game engines.” But it’s the only really open game engine out there, and it’s the only fully open source, royalty-free solution that exists at this time. Of course, there are other game engines, but I see O3DE becoming more important in the years to come, and while it won’t become the one-stop shop anytime soon, it’s bringing us closer in that direction, growing its ecosystem, together with the open source game ecosystem.
Mabel Zhang: So I would offer two perspectives. From a technical perspective, I don’t think it’s possible to have a one-stop shop because there is an inherent trade-off between physics and rendering. Unless you have an infinitely powerful computer, you just cannot get infinitesimal small numbers, and we see that a lot with maritime simulations. How do you simulate dynamics of waves, of water? We don’t even know how to simulate that. And then, if you trade that off with rendering, rendering also requires really, really quick timesteps, which is why the rendering of game engines is so powerful, and when people see that, they immediately say, “This is the best simulator for me, I don’t need any other things.” Whereas, they don’t see the physics, which is what Gazebo decided to prioritize versus rendering.
From an organizational perspective, Open Robotics has been working with NVIDIA, Toyota Research Institute (TRI), and Deep Mine in the past few years to create converters between the formats of all these simulators, which indicates that nobody’s really aiming to be the one-stop shop. We’re trying to convert between the TRI Drake format and Gazebo format, so that we can load files built for another simulator like the game community for Omniverse into Gazebo, and vice versa, so that we can take advantage of each simulator for whatever application they happen to be doing.
Matt Hansen: I don’t think that a one-stop shop is going to happen any time soon, maybe someday in the distant future. The vision of having a unified metaverse will exist, and there will be some all-encompassing simulator that can simulate everything known to mankind, but I think we’re still at least one generation of simulators away from that, possibly more.
Robin Rowe: I think it’s a tough question. We didn’t ask the question of, “Should we even have a one-stop shop?” Is Linux a one-stop shop? That’s kind of what we’re talking about here because, you know, at some point the metaverse becomes the operating system. And so yeah, I guess so.
I think that we’re gonna have these very rich tools, and then we’re going to have specialty tools that could be potentially a very small subset of those features, but are designed to help people with something that’s much more specific. I think it comes out of the tools that people already use as well.
Royal O’Brien: Right. I was talking to someone the other day about the kernel driver and whether or not it’s the core of the operating system. What is the core? What is the one-stop shop? And, you know, the answer is yes and no, because, depending on the application, I can build a Linux core that gives you a complete desktop, or I can give you a completely stripped down Linux core that runs an IoT device. Which one was core? Which one was my one-stop shop? So it’s also a loaded question. I think you’ll never just have one place for everything—there’ll always be something that extends on top of it that makes it into a complete package. So, O3DE is one of the tools in the arsenal to be able to bring robotics to life, Gazebo is another, and there are other elements. It’s the interoperability that makes it possible to pull in all of these tools and use the right tool for the job. And the nice part is, we have a lot of history to draw on with Linux and open source because companies have used them forever.
Adam Dabrowski: So there are many angles to approach when you try to determine which kind of simulation you should use: What kind of efforts should I put into the simulation? What do you want to take from the simulation? What kind of use case do I have outdoors? Do I have an underwater use case? And then, you know, the physics engine will probably be determined by that that suits you. And then again, you might opt to choose two simulators: one that is highly performant, looks pretty, has good sensor simulation, and nice ROS integration; and the other one that is perhaps focused on great physics for this specific thing that your robot does. If it’s aerodynamics, if it’s movement on rough terrain, you can test that separately in the other simulation.
Mabel Zhang: So I would say, in order to pick a simulator for industrial applications, one point is that it needs to be something that is certifiable, it needs to be able to be validated. A second thing I would want is something that can be modular. I can add something to it, and strip down the things I don’t want. If I want to put it on the embedded system, I don’t want to have 3 GB of stuff. A third thing, if i’m an educator, I want a simulator that’s going to lower the education barrier for my students to use. So if it’s cheap, if it’s free, that’s the best. I’ll just leave three points there.
Lars Gleim: I think that, if you want to pick the best simulator for your use case, you should look at the features that you actually need to simulate. Different simulators focus on very different aspects of physics, of simulation, depending on the kind of interaction between the robot and the environment.
For example, do you have a gripper? Do you want to grip deformable objects? Then, if you want to train a machine learning model that is capable of grasping objects in the real world, you actually need to have a very good simulation of DC Form of the physics in the simulator. And the majority of today’s simulators do not support simulating soft bodies, or even deformable objects, well at all. So it really depends on the kind of application.
Let’s say you wanna develop a drone, and you want to validate the aerodynamic properties of that. Probably none of the open source robotic simulators will offer you that kind of simulation right now. So really, look at what kind of scenario you want to simulate, and which of the simulators has the best tool.
Coming back to this question of the one-stop shop, I think what we will see in the future—what we don’t have yet, but which several vendors are working towards, and I also see this as a route for O3DE—is the modularity and configurability to select exactly what you need … where you can enable and disable different aspects of the simulator for your use case. It always depends on your use case. Depending on the stage, the method that you’re using, and what you’re trying to validate, [you need to] choose the best tool for the job.
Royal O’Brien: Right. modularity is definitely a key piece—having different subsystems that you can plug in together in a known framework that you can operate. You want to be able to remove everything out and build up, because unbuilding and unwinding is just as painful, if not more so, than building in many cases.
Adam Dabrowski: So, I’d like to add a few more points to the consideration set when choosing a simulator, which might also be a good introduction to O3DE and its strengths. From my perspective, performance is quite important for simulations. If you want to simulate large areas, lots of robots or sensors that produce a lot of data, then it’s really important for you to be able to run the simulation reasonably fast, especially if you care about real-time simulation, which is useful for many reasons.
In addition, the adherence to a standard is key—so, the interfaces that the simulator provides and how well it’s integrated with ROS. We, of course, do not have simulation interface standards yet—that’s something to produce, hopefully in the near future.
And then another point is, of course, what technology is used, what programming language is used. Do I have to hire a special team, or can I use ROS developers to develop the simulator itself or the modules? So I do prefer C++ and Python programming interfaces and code base.
And then coming to the very important point: the documentation. So I do care about the simulator being well documented, so that I I can find solutions to my issues, solutions to my challenges. And O3DE represents quite a modern approach with a lot of awareness of these requirements.
I also think we have a very attractive renderer, called Atom. We have modular architecture, which is extendable, using modules called Gems. We have what I consider the best kind of ROS integration, in the sense that we write ROS2 code in the simulator, so we can use all the packages—we can integrate them—directly without any bridges, which impacts performance and enables us to communicate directly to the ROS ecosystem and record data efficiently.
When we think about the goals of simulations, for example, presenting your use case to a customer, I think O3DE is quite suitable for this. We actually did that with O3DE and we have a customer project with O3DE right now, so I can speak from experience. It was very good to prototype a robotic simulation quickly with O3DE and ROS2, and we were even surprised at how little effort it took to produce something beautiful that tells a story.
Royal O’Brien: How long did it take you to ramp up and build something on it?
Adam Dabrowski: Just one week, one-person week, I would say. Of course, we are quite experienced doing that. But I think in terms of what we accomplished in just one week, it was pretty impressive.
Lars Gleim: I think Adam made very good points from the developer, or technological, perspective. Of course, that’s not the only perspective, right? So documentation, I think, is always something that can be better. And the onboarding experience with O3DE is also something that’s constantly improving, but that can be easier. So for many newcomers—new companies, new users, new potential contributors—coming to O3DE for the first time, there are a lot of concepts to understand and a lot of pieces to set up to get moving, still.
I think, where we want to go, is that you can use an ecosystem of tools, and O3DE will be one tool within this ecosystem. For example, if you have a warehouse or some production site or an outdoor environment, whatever you want to get started with, you use the best tool to create the digital twin of that environment, and quickly import that into the engine without having to worry about how to set it up properly in terms of all the artists workflows, textures, lighting, and so on. All of this should be fast and easy, with good defaults to get started so that you can directly focus on developing the piece that you need for your customer case, right? That can be simulating the industrial use case that you’re looking at. That can also be implementing the sensor that you’re trying to simulate in the solution. But the key challenge, or goal, not only for O3DE, but for the entire open source ecosystem of simulators, is streamlining the ecosystem and the time that you need until you actually get started working on your specific problem.
Do you think the search for a one-stop solution in simulators is realistic? What goes into your list of considerations when searching for a simulator?
This discussion is an excerpt from an earlier panel discussion. View the live recording of the full discussion here.
