Method Introduction
This agent consists of a low-level skill library and a high-level controller. The low-level skill library focuses on specific aspects of table tennis, such as forehand topspin, backhand aiming, or forehand serve. In addition to training strategies, this study collects and stores information about the advantages, disadvantages, and limitations of each low-level skill both offline and online. The high-level controller responsible for coordinating low-level skills selects the best skill based on current game statistics and skill descriptions.
Furthermore, the study collected a small amount of human vs. human match data as seeds for initial task conditions, including information on position, speed, and spin. The agent is then trained using reinforcement learning in a simulated environment, and existing techniques are used to seamlessly deploy the policy to real hardware.
The agent plays against humans to generate more training data, and as the robot continues to learn, the game standards become increasingly complex, allowing the agent to learn increasingly complex actions. This hybrid "simulation-reality" loop creates an automatic teaching process, continuously improving the robot's skills over time.
Hierarchical Control
Hierarchical control mainly includes the following parts:
- Table tennis playing style: The high-level controller (HLC) first decides which style to use (forehand or backhand);
- Adjustment: Based on statistics from matches against opponents, preferences (H-values) for each HLC are maintained online;
- Selecting the most effective skill: The HLC samples from the shortlisted LLCs based on the adjusted H-values;
- Update: H-values and opponent statistics are continuously updated until the end of the match.
Results
The researchers compared the agent with 29 table tennis players of different skill levels, including beginners, intermediate, advanced, and advanced+ skills. Human players played three matches against the robot according to standard table tennis rules, with slight modifications as the robot cannot serve.
Against all opponents, the robot won 45% of the matches and 46% of individual games. Broken down by skill level, the robot won all matches against beginners, lost all matches against advanced and advanced+ players, and won 55% of matches against intermediate players. This indicates that the agent achieved an intermediate human player level in table tennis rallies.
The robot's inability to beat advanced players is due to physical and technical limitations, including reaction speed, camera sensing capabilities, spin handling, etc., which are difficult to accurately model in a simulated environment.
Playing Against the Robot is Also Attractive
Study participants reported that they greatly enjoyed playing against the robot and gave it high ratings in terms of "fun" and "engaging." They also unanimously expressed that they would be "very willing" to play against the robot again. During their free time, they played with the robot for an average of 4 minutes and 6 seconds out of a 5-minute period.
The Robot Struggles with Backspin
The most skilled participants mentioned that the robot struggled to handle backspin. To test this observation, the researchers plotted the robot's landing rate based on ball spin. The results show that the robot's landing rate significantly decreases when facing more backspin. This deficiency is partly due to the robot's attempt to avoid colliding with the table when handling low balls, and secondly, real-time determination of ball spin is indeed challenging.