1. Introduction
Capable humanoid robots are the most similar to humans and are the result of cutting-edge robotic technology. Independently developing a humanoid robot is non-trivial, and there are still many challenges to overcome before truly autonomous humanoid robots with similar capabilities to humans can be developed. But, research on humanoid robots has been continuously conducted, and attempts to use them in professional performing arts have been made since early 2000. The first robot to officially appear on the stage is Wakamaru, who appeared in ‘I, Worker 2008’, which talks about the coexistence of humans and robots in everyday life (Robertson, Reference Robertson2021). Produced by Mitsubishi Heavy Industries, Wakamaru is a wheeled humanoid robot that moves and performs autonomously on stage. Developed for service purposes, Wakamaru could communicate with humans but was still rather limited in its capabilities. A well-known play called ‘sayonara 2010’ features an android robot named Geminoid F and an actor performing a 20-min monolog (Pluta, Reference Pluta2016). An artificial intelligence robot performer with 16 facial expressions, EveR-3, performed ‘Robot Princess and the Seven Dwarfs 2010’ with a wheel-based humanoid robot, SeRoPi-1 (Lytridis et al., Reference Lytridis, Bazinas, Kaburlasos, Vassileva-Aleksandrova, Youssfi, Mestari, Ferelis and Jaki2019). ‘Night on the Milky Way Train 2013’, noting that it was produced based on Oriza Hirata’s theater theory and that it features four actors and a humanoid robot who interact together to progress the play. During ten days of public performances, the play lasted an hour and a half, and some audience shed tears (Nishiguchi et al., Reference Nishiguchi, Ogawa, Yoshikawa, Chikaraishi, Hirata and Ishiguro2017). The play ‘Spillikin-A Love Story 2015’ tells the love story of Shally, a grandmother who has Alzheimer’s disease and is comforted by Spillikin, a humanoid robot her husband built before his death (Stamboliev, Reference Stamboliev2017). Actors and officials who participated with Myon, a humanoid robot appearing in the musical ‘My Square Lady 2015’, teach him about the special things that make an opera and want to make him a star (Bailey, Reference Bailey2006). Spilikin was played by Robo Thespian, a humanoid robot from Engineered Arts. A humanoid robot named Cornell helps Zoe, who has lost all memory, treat her with the play ‘Cornell 2016’ performed in collaboration with Oriza Hirata. They exchange information through conversation, but Zoe does not recognize Cornell and deletes the records (Jochum et al., Reference Jochum, Vlachos, Christoffersen, Nielsen, Hameed and Tan2016).
Even in dance performances that deliver messages through the body, humanoid robots have been used to move audiences. After practicing for about a year, HRP-4C danced and sang with four choreographers at the 2010 Digital Content Expo (NAKAOKA et al., Reference Nakaoka, Miura, Morisawa, Kanehiro, Kaneko, Kajita and Yokoi2011), and appeared as a model in a fashion show (Kaneko et al., Reference Kaneko, Kanehiro, Morisawa, Tsuji, Miura, Nakaoka, Kajita and Yokoi2011). The humanoid robot HRP-2 performed a 10-min dance in front of an audience of more than a thousand people with hip-hop dancers at a concert hall in Toulouse, France in 2012 (Ramos et al., Reference Ramos, Mansard, Stasse, Benazeth, Hak and Saab2015). Choreographer BLANCA LI uses various types of robots in her performances. Since 2013, she created ‘ROBOT 2013’ where she dances on stage with the humanoid robot NAO in 60 countries (BlancaLi, ROBOT, 2013).
At the 2009 Robot Show, a Toyota partner trumpet-playing robot played trumpet by blowing wind through a mouthpiece just like a human trumpet player (Bloss, Reference Bloss2010). The 2013 Tokyo Maker Faire was dominated by Yuri Suzuki’s Z-Machines, which consists of a guitar, drummer, and keyboard robot (Z-Machines, 2012). In 2013, the robot TeoTronica took turns playing piano, battled with famous Italian pianist Roberto Prosseda and performed a concert with an orchestra. The robot TeoTronica performed live with musicians at concerts, and although not in the form of an exact humanoid robot, it answered questions from the audience or showed facial expressions (TeoTronico THE ROBOT PIANIST, 2014). At Australia’s biggest music festival, Big Day Out, the audience sang and applauded along to the performance of the animatronic robot band Compressorhead (Davies & Crosby, Reference Davies and Crosby2016). Shimon was a marimba-playing robot created by Gil Weinberg in 2017, that played alongside musicians while imitating human gestures (McKenzie et al., Reference McKenzie, Thompson, Moore and Ben-Ary2021). In Keiichiro Shibuya’s ‘Scary Beauty 2018’, Alter 2 sings while conducting an orchestra (Sigman, Reference Sigman2020). Bosnia’s well-known rock band Dubioza Kolektiv performed with Robby Megabyte at the 2020 Quarantine Show (Meet Robby Megabyte, Bosnia’s first robot rock band musician, Reference Megabyte2021). In various performing arts such as plays, musicals, concerts, and dances, the frequency of humanoid robots is increasing, and their role is gradually expanding (Bogue, Reference Bogue2022).
Magic is a popular-friendly art among the performing arts and has a long history. People consider the first magician of humankind to be the ancient Egyptian magician Dedi 2500 BC, and when they look at the paintings on the walls of the tomb of Baqet III in the 21st century BC, they think of cup and ball magic. Documented records indicate that the Acetabularii group has practiced using cups and stones for magic for about 250 years. People considered magic witchcraft and persecuted it for a time, but it was a form of entertainment for the masses by the 18th century. Jean Eugène Robert-Houdin, the father of modern magic, founded the Magic Theater in France in 1845 and John Henry Anderson in London in 1840. They made the spectators feel willing to pay for the magic to be worth the price. Harry Houdini, who is best known, is a representative escape magician who developed magic tricks using the stage (The History of Magic, 1978).
Magic tricks have become more professional due to the rapid development of science and technology. Online media has made it easy for anyone to learn magic tricks, which has increased audience expectations for magic. Magicians use the latest scientific principles and techniques that have not been popularized to create more sophisticated and creative magic. Magicians use science and technology to make things impossible in reality seem possible through magic (Bailey, Reference Bailey2006).
The humanoid robot application challenge (HAC) has emerged as one of the leading international performing arts that brings together humanoid robot researchers worldwide. In addition, HAC-magic and music is an international competition that features intelligent humanoid magicians performing magic shows using cutting-edge technology. The HAC started at the IEEE International Conference for Robotic Systems (IROS) in 2011, and HAC-Magic and Music began to be showcased in 2017. Music was added in 2018 to encourage musicians to participate. Due to the COVID-19 pandemic, the robot magic show occurred virtually in 2020 and 2021. Humanoid robot magic shows between 2017 and 2021 are comprehensively analyzed in this study.
Another reason for the interest in magic is that attention, which is crucial to human intelligence, is today still a poorly understood and researched topic in artificial intelligence research. Magic, on the other hand, is at heart the art of skillfully capturing and misdirecting the attention of the audience. Therefore, it is an excellent domain for investigating attention in robots and humans.
The following Section 2 explains the history and meaning of the HAC, the elements that make up the show, and the types of magic performances. Section 3 summarizes the hardware platforms of robot magicians who have participated in the robot magic show since 2017. Section 4 describes the robot show in detail, such as types of magic performance, magic tricks, tools, algorithms, and software frameworks. Finally, we present our conclusion in Section 5.
2. Humanoid robot application challenge: robot magic and music
The humanoid application challenge is a fascinating competition that seeks to push the boundaries of what is possible with humanoid robots, by combining robotics and magic to create new forms of entertainment. This competition challenges participants to develop humanoid robots that are capable of performing a wide range of magic tricks, requiring a combination of accurate perception, complex manipulation, and efficient human–robot interaction (HRI). International robot competitions like the HAC have gained significant attention among robotics researchers, as they provide an excellent platform for testing and evaluating new technologies and approaches in robotics. These competitions offer an opportunity for researchers to showcase their latest developments, and to compete against other experts in the field, in order to drive innovation and accelerate progress in the field of robotics. But the HAC is more than just a research benchmark or a technical challenge—it is also a source of fun and entertainment for the audience. Humanoid robots are particularly well-suited to this type of entertainment, as their body structure, walking gait, and sensory interface are similar to those of humans. It makes them more appealing and exciting to watch and allows the audience to connect with the robot in a way that would not be possible with other types of robots. By combining robotics and magic in this way, the HAC is accelerating the development of multimedia content that is engaging, interactive, and fun for all ages. It is inspiring the next generation of robotics researchers and magicians and pushing the boundaries of what is possible with these remarkable machines. As the competition continues to evolve and grow, we can expect to see even more exciting and innovative uses of humanoid robots in entertainment, education, and beyond.
The humanoid robot magic competition, known as the Humanoid Application Challenge, was first launched in 2016 with the aim of pushing the boundaries of what is possible with humanoid robots in the context of magic tricks (Moon et al., Reference Moon, l. Moon, Sun, Baltes and Kim2017). The competition challenges participants to develop humanoid robots that can perform magic tricks like human magicians and communicate effectively with the audience. The HAC has become an important event for the robotics and magic communities, attracting participants from all over the world who are eager to showcase their innovative and creative solutions. The competition provides an opportunity for participants to demonstrate their expertise in areas such as computer vision, machine learning, and HRI. One of the key goals of the HAC is to inspire a new generation of roboticists and magicians to work together to create new and exciting forms of entertainment. The combination of advanced technology and magic tricks principles creates a powerful synergy that is driving innovation in both fields. In addition to inspiring innovation and collaboration between roboticists and magicians, another aim of the HAC is to challenge participants to develop humanoid robots that can perform magic tricks like human magicians, and communicate effectively with the audience. The robots must also be able to communicate effectively with the audience, creating a sense of engagement and interaction that is essential to the magic experience. Magic has always been a fascinating form of entertainment, and the use of science and technology to create illusions has added a new dimension to the art of magic. Many of the magic tricks used in shows are based on mathematical principles such as probability, geometry, and physics. On the other hand, science-based magic tricks use concepts from various scientific fields like chemistry, physics, and psychology to create stunning visual effects that leave the audience awestruck. In recent years, technological advancements have revolutionized the art of magic. Magicians now use technology like robotics, augmented reality, and virtual reality to enhance their performances and create even more stunning illusions. As robots become more advanced, the possibilities for robot magicians to create new and exciting magic tricks are endless. Incorporating magic and science to create mind-blowing experiences, robot magicians can captivate audiences across the globe, and entertain them for years to come. The HAC competition not only challenges participants to develop humanoid robots that can perform magic tricks but also focuses on optimizing complex technologies-based magic tricks for the robot magician to interact with the audience effectively (Williams & McOwan, Reference Williams and McOwan2014). The competition requires participants to apply various technical and scientific principles to develop their robot magicians. From the judgment viewpoint, the winner is decided based on several factors, including successful performance, HRI, and a positive scientific contribution applied to the robot. A robot magician that can perform successful magic tricks and engage with the audience through effective HRI while demonstrating a positive scientific contribution to the field is more likely to win the competition (Jeong et al., Reference Jeong, Yang and Baltes2022). This approach not only encourages the development of advanced robotic technology but also promotes the importance of scientific contributions in the field of robotics.
The Robot Magic show provides a unique opportunity for the audience to witness and interact with humanoid robot magicians from all over the world. It’s not just a competition; it’s a performance art that offers a different kind of appreciation for the audience. By being part of the show, the audience can learn about the entire process of preparing and conducting the show. They can listen to the explanations of the participants, gain insight into the preparation of the show, and witness a new viewing environment. In this challenge, participants are researchers who study humanoid robots, show planners, magicians who analyze and create magic tricks, light and sound engineers, and spectators to other participants’ performances. By performing different roles in the planning and execution of the event, they work together to create a unique and memorable experience for the audience. To participate in the HAC challenge, interested individuals or teams must write a detailed 2–3 page outlining their robot magicians and team members, including descriptions of the magic tricks they plan to demonstrate and post a video of their magic show demonstration online. Only those who have passed the preliminary round are eligible to participate in the finals. The challenge lasts an average of three nights and 4 days, during which time each team must complete the final within the preparation time of 5 min, the magic show of 10 min, and the presentation of 10 min. To ensure fair competition, information on magic tricks is disclosed to all participants, as the humanoid robot must explain all the processes until the magic is implemented. While it is forbidden to disclose magic tricks in the field of magic, the HAC challenge encourages participants to create new magic by revealing the tricks used in the competition. Therefore, many teams in the entry stage start by purchasing magic tools or mimicking existing magic shows. However, the ultimate goal is to create a unique and new magic show where humanoid robots and humans can come together naturally, showcasing the best of science and magic.
Magic is a fascinating art form that has evolved over time into various types of magic. These include illusion magic, close-up magic, parlor magic, and stage magic, each with its unique characteristics and techniques. Close-up magic involves performing tricks within 3 m of the audience, using small everyday objects such as coins or keys. Due to the proximity of the audience, close-up magicians require a high level of skill and expertise. Parlor magic, also known as platform magic, is performed for a larger audience in a spacious area without a set stage. Magicians performing this type of magic focus on tricks that are less likely to reveal their secrets due to the open space. Parlor magic is suitable for an audience size between close-up and stage magic, and it allows the magician to interact with the audience while performing. Stage magic is performed on a formal stage in a performance hall equipped with lighting and sound. This type of magic is usually performed by professional magicians, and it involves large-scale illusions, such as making objects disappear or levitate, and includes the use of elaborate props and special effects. Illusion magic is a type of stage magic that requires specialized equipment and an assistant. It involves creating large-scale illusions that change the perception of space or make large objects disappear, such as the famous ‘evaporating the Statue of Liberty’ trick. Only a handful of magicians worldwide can perform grand illusions due to their complexity. The HAC is a cutting-edge competition that merges the art of magic with technology by incorporating humanoid robots into magic performances. The challenge is focused on creating unique and captivating experiences for audiences through stage magic and close-up magic performances featuring robot magicians. With the help of advanced technology, HAC aims to push the boundaries of traditional magic performances by introducing robots as assistants and main attractions, bringing a fresh and innovative approach to the world of magic. These performances offer audiences a truly unforgettable and immersive experience that combines the best of magic and technology.
3. Robot magicians
The concept behind HAC, which started in 2011, was to provide teams with the greater creative freedom to showcase their skills compared to other conventional robot competitions such as Robocup and FIRA Roboworld Cup. Since 2016, the HAC has been organized every year, featuring impressive magic tricks performed by high-performance robots. The competition’s focus was shifted to robot magic as the main theme. Performing magic tricks is an art that involves utilizing tricks to astound audiences with seemingly impossible or peculiar phenomena. It presents a formidable challenge for humanoid robots and therefore is clearly one of the intriguing areas for research. This section examines the specifications of the humanoid robot platforms that functioned as robot magicians for the HAC between 2017 and 2021. HAC—Magic and Music featured an assortment of humanoid robots, including commercially available models and those designed and developed by the participants themselves. The following Figure 1 illustrates the robot models of humanoid robot magicians that have officially participated in the competition over the years, including Thormang3, Baxter, Darwin-OP series, Sweaty, Robinion series, Bioloid, and ARC Humanoid Robot.
Humanoid robot platforms that participated in HAC, from left to right: Sweaty, Thormang3, Baxter, Robinion2, Robinion, ARC Humanoid, Darwin-OP3, Darwin-OP2, Darwin-OP, and Bioloid.
Designing and developing humanoid robots is a complex and challenging task that requires expertise in multiple fields such as mechanics, control, perception, and cost efficiency. That is why careful planning and execution, and a significant investment of resources. Nonetheless, several teams at HAC who created their own robots, namely Sweaty, Robinion Series, and ARC, demonstrated noteworthy magic tricks and technical expertise. Except for the teams that developed the robots described above, the teams competed in the competition with commercial robots. Representative companies worldwide that produce intelligent humanoid robots commercially include Agility Robotics, ROBOTIS, Aldebaran Robotics, Rainbow Robotics, Vstone, and UBTECH Robotics. Especially, user-friendly products of ROBOTIS are favored by numerous roboticists for conducting research in diverse areas of robotics, and the products are widely used by most participants in the competition. ROBOTIS manufactures several humanoid robots, including Thormang3, Darwin-OP, and Bioloid, as well as smart actuators named Dynamixel, which are used by Sweaty, Robinion, and ARC. ROBOTIS has contributed significantly to advancing the field of humanoid robotics by introducing accessible solutions to complex challenges. For instance, they have released the Darwin-OP series of open-source-based robots, which stand at approximately 50 cm in size and offer a user-friendly platform for robot development (Ha et al., Reference Ha, Ha, Tamura, Asama, Han and Hong2011). In addition, they have developed the Thormang3, a human-sized robot capable of performing intricate tasks in the Darpa Robotics Challenge. ROBOTIS also offers various robot platforms, including the Bioloid and Engineer kits, which allow junior students to operate and experiment with robotics. Since its launch by Heartland Robotics in Germany in 2013, Baxter has been a popular choice in both academia and industry for its ability to collaborate with humans on a variety of tasks (Fitzgerald, Reference Fitzgerald2013). This industrial cooperative robot is highly versatile and comes at a reasonable price, making it a valuable asset for many organizations. Sweaty is a versatile robot developed for autonomous soccer and other tasks by researchers at the University of Offenburg in Germany (Schnekenburger et al., Reference Schnekenburger, Scharffenberg, Wulker, Hochberg and Dorer2017). In addition to soccer, Sweaty is also capable of performing tasks such as magic and chess. The ARC Humanoid Robot is a small-sized humanoid robot developed at the University of Tabriz in Iran (Saeedvand et al., Reference Saeedvand, Aghdasi and Baltes2020). It is manufactured using open-source-based 3D printing technology and can be used as a research platform for various applications. The robot is equipped with a stable walking gait based on closed-loop control and is available at half the cost of commercially available robots. Team ZSTT has been developing the Robinion series of intelligent humanoid robots since 2013 (Jeong et al., Reference Jeong, Yang and Baltes2020, Reference Jeong, Yang and Baltes2022). These robots are designed to be multifunctional platforms for conducting research in several areas, including closed-loop-based stable walking gait, balance control in highly dynamic environments, and manipulation. In addition to research, the Robinion series is also capable of providing entertainment through robot magic shows and participating in competitions such as Robocup and FIRA as robot athletes.
Table 1 shows the various types of humanoid robot platforms that participated in the HAC competition from 2017 to 2021. HAC is interdisciplinary research fused with performances such as engineering technology, acting, and art specialized for humanoid robots, and there are around 10 teams registered annually, of which approximately six teams qualify for the finals. While the number of teams making it to the finals each year is practically consistent, there has been increasing diversity in the platforms used since 2017. In that year, all teams except for Robinion used the Darwin-OP series, but in subsequent years, teams showcased their own in-house developed robots such as Sweaty, Polaris, ARC, Robinion, and Robinion2. This helped to attract more attention to the competition and move away from the commercial robots typically used in magic shows. With the participation of high-performance humanoid robots such as Thormang3 and Baxter, the range of magic tricks performed has diversified, and the technical level of the competition has increased.
Humanoid robot platforms as magicians participating in the HAC since 2017
The performance of humanoid robots is directly impacted by the design of their exoskeleton and the selection of electronic components. Thus, it is crucial to carefully consider the robot’s purpose when determining the design direction and component specifications. For instance, teams competing in the DARPA Robotics Challenge placed emphasis on proper manipulator design to perform complicated manipulation tasks, whereas those in the RoboCup humanoid league focused on leg design for a speedy walking gait, powerful kicks, and robustness against falls. To provide further insight into the mechatronics specifications of humanoid robots in the HAC competition, Table 2 outlines information such as their height, weight, degrees of freedom, actuator and sensor types, frame materials, and computing units. The Thormang3, Baxter, and Darwin-OP series of humanoid robots are readily available for commercial purchase. Each of these robots has been carefully designed and manufactured by their respective companies, which have taken great care in selecting exoskeleton designs and electrical components such as controllers and sensors that are specifically tailored to the needs and capabilities of humanoid robots. ROBOTIS designed and developed the Thormang3 humanoid robot to provide a platform for research and development in the field of humanoid robotics. The Thormang3 is a highly advanced robot that can be programmed and customized for a wide range of applications, including education, research, and entertainment. In addition, the Thormang3 incorporates advanced sensors, actuators, and control systems that enable it to perform a wide range of tasks and movements with precision and accuracy. The Baxter is an industrial collaborative robot with a series elastic actuator that is designed to work alongside human workers in industrial settings, such as manufacturing plants and assembly lines. The Baxter has a unique design that allows it to perform a wide range of tasks, including picking and placing objects, sorting and packing items, and operating machinery. The robot has two arms that can be programmed to move and manipulate objects with precision, and its sensors and cameras allow it to detect and adapt to its environment in real time. One of the key features of the Baxter is its ease of use. The robot is designed to be programmed by non-experts, using a simple graphical user interface that makes it easy to teach the robot new tasks and movements. This makes the Baxter a versatile and flexible tool for a variety of industries and applications. The Darwin-OP series is also designed for a variety of applications, including research, education, and entertainment. These robots are highly customizable and can be programmed to perform a wide range of tasks and movements using a variety of sensors and control systems. The Darwin-OP series is often used in research settings to study HRI and to develop new algorithms for motion planning and control in humanoid robots. The Darwin-OP series is a highly efficient platform for humanoid robotics that supports open-source hardware and software. This robot series has made significant contributions to the field of humanoid robot research, inspiring new developments and advances in the industry. The Sweaty was developed for playing the autonomous soccer competition and then expanded into developing a highly versatile arm and gripper with a large range of motion for diverse research applications, leveraging their advanced technological expertise. Moreover, they designed the robot structure using a linear actuator, which differs from the standard kinematics structure typically used for humanoid robots. The ARC Humanoid Robot is an innovative humanoid robot that was entirely 3D printed, making it the first of its kind. This robot was developed as an open-source project, allowing anyone to build a small-sized, high-performance humanoid robot. The Robinion series is a humanoid robot that is a size between the Thormang3 and Darwin-OP series but with a higher degree of freedom and an added gripper. This makes it a versatile, multifunctional robot that can be used for a range of applications, including HAC, FIRA, and Robocup competitions. The robot has advanced sensors and algorithms that enable it to perform complex tasks, such as object recognition and manipulation, with precision and accuracy. The Robinion series has been designed with a focus on autonomous and lightweight mechatronics. With its sophisticated technology and compact design, the Robinion series is one of the numerous advanced humanoid robots. Robot magicians primarily use Dynamixel actuators from ROBOTIS Inc., which are cost-effective and can be developed efficiently. The robots use an Intel CPU as their computing unit, which is capable of supporting various operating systems. A sub-controller is also included to control the actuator. These humanoid robot platforms use sensors like cameras, IMUs, F/T, and lidar to autonomously perform given tasks.
Mechatronics specifications of humanoid robot magicians
4. Analysis for robot magic shows
Humanoid robots are increasingly being used in the entertainment industry, particularly for magic shows. The HAC evaluates the convergence of robotics technology and magic tricks while also darting at the direct entry of humanoid robots for collaboration with the entertainment industry by performing a comprehensive evaluation of performances like real magic shows and dramatic elements through music. It is clear that the dramatic elements and performance enhancements of humanoid robots, since their first show of robotic magic in 2016, are increasingly becoming a realization in real-world entertainment. The competition leads in performing a comprehensive evaluation of performances such as real magic shows and dramatic elements through music. The competition also helps to enhance the audience’s experience with robotics. One notable example of the use of humanoid robots in entertainment is the team ZSTT, who won the robot magic competition in 2018. A Korean professional magician participated as a director for the team’s show and assisted with the story of the robot magic show, the types of magic, and the interaction between the assistant and the robot. The show was a touching performance that demonstrated the potential of humanoid robots in the entertainment industry (Jeong et al., Reference Jeong, Yang and Baltes2020). Nevertheless, the range, speed, and accuracy of robot movement are insufficient for the audience to feel, and there are several areas for improvement. To improve the performance of humanoid robots in magic shows, it is necessary to analyze previous robot magic shows and suggest possible directions for use in the entertainment industry in the future. In Table 3, we illustrate the types of magic performance, magic tricks and tools, and algorithms and frameworks applied to the humanoid robot platforms of the teams participating in the HAC from 2017 to 2021 in detail.
Analysis of key points of teams participating in HAC
Each of the items in Table 3 is analyzed in-depth in Sections 4.1 and 4.2. The ultimate goal is to create an immersive and engaging experience for the audience that combines the dramatic elements of magic with the technological advancements of humanoid robots.
Figure 2 provides valuable insight into the growth and diversity of the HAC robot magic show since its inception in 2017. The chart numerically analyzes the number of humanoid robot platforms, sensors used in magic tricks, and magic tricks that have participated in the competition over the years. In the early days of the competition, the robot magic show predominantly used the Darwin-OP, a commercial humanoid robot, as the primary platform. However, since then, the competition has witnessed an increase in the variety of robots used, such as Robinion, Sweaty, Baxter, and Thormang. This indicates the growth and progress of the competition in terms of attracting and accommodating diverse humanoid robots which can perform magic tricks and communicate effectively with the audience. Additionally, the chart highlights the growth in the number of magic tricks and the number of sensors used in magic tricks, which is essential for developing robots capable of performing more sophisticated and advanced magic tricks in the future. These numerical analyses demonstrate the continuous evolution of the HAC robot magic show, encouraging participants to push the boundaries of humanoid robots’ capabilities and perform novel and exciting magic tricks.
The analysis chart for the growth and diversity of the HAC robot magic show since 2017 (Purple: number of magic tricks, Green: number of sensors, Blue: number of robot platforms).
4.1. An overview of magic tricks
Performing a successful magic show is not just about executing fantastic magic tricks but also about engaging the audience and creating a sense of drama and suspense. To achieve this, the magician must carefully craft their performance to include a range of elements that work together to create a captivating experience. One important aspect of a magic show is the use of visual effects. These can range from simple card tricks to complex illusions involving levitation, disappearing objects, or even people. The key to making these visual effects effective is to make them unexpected and unpredictable, keeping the audience on the edge of their seats wondering what will happen next. Another important element is communication between the magician and the audience. A skilled magician knows how to use humor, storytelling, and audience participation to create a sense of connection and engagement with the audience. This not only makes the show more enjoyable for the audience but also helps to build anticipation and excitement as the performance unfolds. To enhance the dramatic effect of the magic tricks and communication, music is also a crucial component. The right music can help to set the mood and create an emotional connection with the audience. For example, suspenseful music can be used to build tension and excitement during a particularly tricky trick, while a more upbeat tune can be used to create a sense of fun and playfulness during a lighter moment. When all of these elements are combined effectively, a truly suspenseful and captivating magic show is created, leaving the audience in awe and wondering how the magician pulled off such incredible feats.
The HAC is an annual competition that focuses on showcasing the capabilities of humanoid robots and promoting their use in various industries, particularly in entertainment. Until 2018, the competition primarily focused on showcasing magic tricks performed by robots and their human assistants, as well as demonstrating how these robotic performers could interact with humans on stage. The competition was designed to highlight the potential of humanoid robots as a tool for enhancing live shows and performances. In 2019, HAC expanded the scope of its competition by adding a musical element. The competition now showcases the musical abilities of humanoid robots, such as singing, playing musical instruments, and even conducting an orchestra. This addition reflects the growing interest in using humanoid robots in the music industry, particularly in live concerts and performances. By including a musical component, HAC aims to demonstrate the versatility of humanoid robots and to provide a platform for musicians and performers to explore new possibilities for their art.
The HAC at the IROS has always been an exciting event for roboticists. Prior to the pandemic, the competition included a robot magic show, where roboticists demonstrated their creations by performing magic tricks on stage. However, the roboticists and robot magicians in the show sometimes made it seem awkward and unpolished. Nevertheless, these unexpected moments of imperfection often added an element of drama and entertainment to the show. The competition’s shift to online due to the pandemic led to changes in the format of the magic show. The camera for online streaming was installed close to the stage, enabling participants to use small magic tools and showcase a variety of magic tricks that would not have been possible on a larger stage. The use of small magic tools and the intimate setting also highlighted the precision and dexterity of the robots and the creativity and ingenuity of their creators. However, the smaller stage also presented limitations, especially for large-scale magic performances.
Magic shows are incomplete without the classic trick of card magic, which has remained a crowd favorite for years. Interestingly, even in humanoid robot magic shows, card magic tricks hold a significant position, accounting for almost 40It’s no surprise that the audience finds card magic so intriguing, as it often reminds them of magic performances they have seen on TV, increasing their anticipation for what’s to come. Teams TH Song (2021) and Snowbots (2017) have won competitions by wowing the judges with their excellent performances featuring card magic as their main trick. Similarly, Team Liquid Motion (2017, 2018) also impressed the audience with their card magic performances, leaving everyone spellbound. However, some teams have chosen to incorporate other magic tools to create a unique experience for the audience. Team ZSTT (2018) presented a magical show with the theme of ‘failure and success’ using various tools such as flying silk, sketchbook, bottle & marble, along with non-verbal communication between the robot magician and assistant. In the same year, the Sweety team demonstrated their mastery of manipulator tasks with a cups and balls magic trick, showcasing a deep learning algorithm that precisely locates the cups to present the high ability of the robot. In 2021, team Robot Magic Skit presented the first black box magic trick using a thermal imager, while Team ZSTT demonstrated the mathematical logic behind a magic trick, ensuring its scalability. The Thormang-wolf team thrilled the audience in 2020 by showcasing a sharp nail magic trick with constant interaction that kept the spectators on edge until the end of the performance. In 2019, Team NTNU-ERC caught the audience’s attention by performing a mind-reading magic trick using a coin. Team Snowbots also created a novel form of the magic show by combining magic tricks with a robot rock band for the first time. Every year, the teams participating in the humanoid robot magic show unveil new and innovative magic tricks using props such as a spinner, magic box, cup, rope, cup and sponge ball, recorder, can, and magic wand, providing endless entertainment and delight to the audience.
4.2. A technical skill of magic tricks
The HAC has brought about a new generation of robot magicians that are capable of performing a wide range of magic tricks. These humanoid robots are equipped with high degree of freedom joints that allow them to move freely and perform intricate actions with remarkable precision. Additionally, they are equipped with various sensors, including a camera, microphone, and IMU, which enable them to detect and respond to their surroundings in real time. To perform magic tricks that involve interaction with the audience, the robot magicians have to recognize the current state of the environment and take appropriate actions autonomously. This requires them to process a large amount of data from their sensors and develop sophisticated algorithms that allow them to react quickly and accurately to changing conditions. For example, if a robot magician is performing a card trick, it must be able to recognize the cards being held by the audience and respond accordingly. Similar to human magicians, robot magicians use sight and hearing to communicate with the audience and show off their magic tricks. However, the way they achieve this is quite different. Robot magicians use advanced image recognition and voice recognition technologies to identify objects and interact with the audience differently than relying on the senses of human magicians. In other words, robot magicians use its camera to detect a particular object in the audience and then use its microphone to ask the audience member to perform a specific action with the object.
About 80% of the robots that participated performed interesting magic tricks which involved the robots recognizing and manipulating various objects such as cards, cups, balls, post-its, and even human faces using advanced image processing algorithms, machine learning, and deep learning techniques. This impressive technique was made possible by the robots’ ability to detect and analyze visual data, such as images and videos, and make sense of them using complex algorithms. These algorithms allowed the robots to not only recognize specific objects but also to understand their spatial relationships and how they interact with one another. This enabled the robots to perform various magic tricks, such as guessing a card that a human had chosen. Performing magic tricks with humanoid robots involves precise movements and interactions between the robot and the objects or people involved in the trick. For example, magic tricks such as Nail, Bottle & marble, and Mind-reading require the robot to make precise and coordinated movements in order to achieve the desired effect. To achieve these movements, researchers use techniques such as forward and inverse kinematics solvers, which allow them to calculate the necessary joint angles and positions for the robot to perform the required actions. These solvers take into account the physical constraints of the robot’s joints and limbs, as well as the desired position and orientation of the objects being manipulated. In addition to these complex movements, simpler motions such as greetings and other expressive gestures can also be used to add excitement and engagement to the magic show. These pre-programmed motions are designed to mimic natural human movements and can be triggered by specific cues or gestures from the audience.
In recent years, teams participating in the HAC competition have been pushing the boundaries of what is possible with humanoid robots and magic shows. Some of the most impressive innovations have come from teams such as NTNU-ERC in 2019 and Thormang-wolf in 2020, who used point cloud algorithms using Lidar to enhance their magic show’s technical capabilities. These algorithms allowed the robots to create three-dimensional representations of their surroundings, enabling them to better understand the objects and people around them and interact with them in more sophisticated ways. In addition, other teams, such as ZSTT, Snowbots, and SRM Humanoids, focused on using voice recognition algorithms to interact with the robot magician, audience, and assistant. These algorithms allowed the robots to recognize and respond to specific commands and questions, as well as engage in more natural and fluid conversations with humans. With these advanced technologies, the teams were able to enhance their magic shows and make them even more engaging and impressive. At the same time, the robot magicians were able to interact with the audience and assistants in more dynamic and natural ways, creating a truly immersive and entertaining experience.
Robotics researchers and engineers are constantly working to develop more sophisticated and capable robots, and many of these robots are built using the robot operating system (ROS), an open-sourced meta-operating system. This powerful framework provides a wide range of tools and capabilities that enable roboticists to design, build, and test robots more quickly and effectively than ever before. One of the key benefits of using ROS for magic shows is its modularity. The system is designed to be highly modular, which means that different components of the robot can be easily reused, reinvented, and extended as needed. This allows developers to build complex systems quickly and efficiently, without having to reinvent the wheel every time they want to add a new feature or capability. In addition to its modularity, ROS also provides an integrated package for logging and visualization, which can be incredibly useful during the development process. This package allows developers to easily monitor and debug their code, identifying issues and fixing them more quickly than they would be able to otherwise. The use of ROS in robot magic shows has become increasingly common in recent years. The framework provides a level of convenience and flexibility that was previously unavailable, allowing developers to create more diverse and engaging magic tricks that showcase the capabilities of these remarkable machines. As the field continues to evolve, we can expect to see even more innovative uses of ROS and other advanced technologies in robot magic shows, pushing the boundaries of what is possible with these incredible machines. In addition to ROS, some teams participating in the HAC competition have developed their own frameworks for building robot magicians. For example, Robinion of Team ZSTT used the ZSTT-framework, which was developed in-house by the team. This framework was designed specifically for building the Robinion that could perform magic tricks and provided a range of capabilities that were tailored to the needs of the team. Similarly, other teams using Darwin-OP and OP2 used the OP-framework, which is based on open-source software. By using these custom frameworks, teams were able to build robots that were specifically tailored to the needs of their magic show. These frameworks provided a range of capabilities that were optimized for the types of tricks that the teams wanted to perform, allowing them to create more complex and engaging performances that showcased the full capabilities of their robots.
5. Conclusions
The purpose of this study is to analyze and evaluate the performance of humanoid robots that participated in the HAC, specifically as the main theme of magic tricks and technologies. In order to better understand the technical aspects of the robot magic show, which is a performing art that has evolved, we subdivided and explained the various elements that make up the humanoid robot magic show based on the history of magic. To achieve our research objectives, we conducted an in-depth review of the hardware and magic shows of humanoid robot magicians who participated in the challenge from 2017 to the present. Through algorithmic analysis of the robot magicians used in the magic show, we were able to identify recent trends and advancements in humanoid robot development. Additionally, we analyzed the magic tools and performance styles used in the show to gain a better understanding of how magic is incorporated into the performance. Our analysis revealed that there are few instances where technical research has been conducted on the use of humanoid robots in the performing arts, particularly in the realm of magic. Nonetheless, our study confirms that the convergence of technology and entertainment has the potential to contribute significantly to the robotics and entertainment industries. The Humanoid Robot Magic Show is a unique performance art that brings together engineers, assistants, and audiences in an interactive setting centered around humanoid robots and magic. The convergence of fascinating technology and art will contribute to the robotics and entertainment industries.
Acknowledgements
This work was financially supported by the ‘Chinese Language and Technology Center’ of National Taiwan Normal University (NTNU) from The Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education in Taiwan, and Ministry of Science and Technology, Taiwan, under Grants no. MOST 111-2918-I-003-003-, MOST 110-2923-E-003-001-MY3, and MOST 110-2221-E-003-023. We are grateful to the National Center for High-performance Computing for computer time and facilities to conduct this research.
