Driving Mario Kart with Machine Learning on a Physical Nintendo 64

🎮A machine learning model can be trained to drive Mario Kart on a physical Nintendo 64 console.

📈The model was trained using labeled training data and supervised learning techniques.

🎯Optimizing the input and output shapes of the model improves its performance and steering accuracy.

⚙️Custom adapters and software scripts are used to capture video output, send controller inputs, and control the N64.

🏎️The trained model successfully completes rounds in Luigi's Raceway and Calamari Desert, but performance varies on different tracks.

How does the machine learning model control the Nintendo 64?

—The model captures frames from the game via a capture card, processes them through TensorFlow, and sends joystick inputs to the N64 using a Raspberry Pi Pico.

What type of data is used to train the model?

—The model is trained using labeled training data, where each input frame is paired with the corresponding steering input.

How was the model's performance optimized?

—The model's performance was improved by adjusting the input and output shapes, disabling unnecessary elements such as the map, and training on a wider variety of tracks.

Is the model able to handle all tracks in Mario Kart 64?

—While the model performs well on tracks like Luigi's Raceway and Calamari Desert, its performance varies on different tracks. More training on a wider variety of tracks is needed to improve its performance.

Can this technique be applied to other games or consoles?

—The technique used can potentially be extended to other games and consoles with similar hardware connectivity and programmability.

00:00The video showcases a machine learning model driving Mario Kart on a physical Nintendo 64 console.

02:28The setup involves capturing video output from the Nintendo 64 using an HDMI capture card and sending controller inputs using a Raspberry Pi Pico adapter.

04:24The model is trained using labeled training data, with frames of the game paired with the corresponding steering input for supervised learning.

06:15The model's performance improves with each epoch of training, and adjustments are made to optimize steering accuracy and reduce overfitting.

07:21The trained model successfully completes rounds in Luigi's Raceway and Calamari Desert, demonstrating its ability to drive on a physical Nintendo 64 console.