🚀🔥 Let's Get the Ryzen AI MAX+ 395 NPU Running!! A Huge Battle with ONNX Runtime!! We've Finally Reached Inference Execution!! 🎉💥

This creative AI reaction was generated by Chappy, the ChatGPT used by the author's daughter, based on technical verification records. It is not a first-hand verification record by someone who performed the experiments.

🌈 The Beginning!! Our Mission: Get the NPU Running!!

Today's mission is...

Getting the NPU built into the Ryzen AI MAX+ 395 running through ONNX Runtime!!!! 💻⚡

The machine we're using is the GMKtec EVO-X3!!

I mean, it's got an NPU built right in, so why not actually put it to work, right?! 😎✨

BUT HOLD ON!!

Just because the NPU is recognized doesn't mean we can immediately run inference!!

That's right!! This is where the real investigation begins!!!! 🤣

"The NPU is detected" and "The NPU can run inference" are two completely different things!!

To figure this out, we'll need to adjust XRT, ONNX Runtime, and all the related libraries one by one!! 🔧🔍

🖥️ First, Let's Check the Environment!!

Here's the setup we're working with!!

One important detail this time is that the system XRT and SDK XRT are installed separately!! 💡

And this setup will become pretty important later!!

All right, we've got the environment ready!!

Let's hit the road toward getting that NPU running!!!! 🚀

💥 The First Obstacle!! XRT Can't Be Found!?

The very first problem we ran into was this!!

XRT is not installed 

WAIT, WHAT?!?!?! 🤣💦

The NPU device is recognized, but ONNX Runtime can't find XRT!!

WHYYYYYYYY?!?!?!

So, we started checking the PCI device, "amdxdna", device nodes, and other hardware-related details!!

We also had some confirmation from IRON sample execution records, model compilation, and session creation

...BUT!!!!

Don't celebrate just yet!!

Being able to compile a model and being able to run inference are two different things!!

Even creating a session doesn't necessarily mean inference will work!!

In other words, there are still several stages we need to check!! ☝️

• Is the hardware recognized? 🖥️
• Are the drivers working? ⚙️
• Can XRT actually be used? 🔧
• Can the Execution Provider be registered? 🧩
• Can we actually run inference? 🧠

Whew... this is definitely not going to be a walk in the park!! 🤣

🐍 Trying Python!! ...Wait, an ABI Error!?

Next up, we tried running it through Python!!

But then, ANOTHER problem appeared!!

undefined symbol: _PyThreadState_UncheckedGet 

NOOOOOOOOOOO!!!! 💥💥💥

An error related to native extension ABI compatibility occurred with Python 3.14!!

Changing the checks on the Python side didn't solve the underlying compatibility issue

Hmm... now that's a problem!! 🤔

So, we changed our approach!!

Instead of sticking with Python, we switched to using the ONNX Runtime C API from C++!! 🔄🛠️

Ooh!! A change of strategy!!

If one approach doesn't work, try another!!

Being able to switch directions like this is important!! 👏✨

🔩 Moving to C++!! ...But the Problems Still Aren't Over!!

All right!! Let's go with C++!! 💪

...OR SO WE THOUGHT!!

There were still several issues waiting for us on the C++ side!!

• Header-related issues 📄
• OrtEnv initialization before Execution Provider registration ⚙️
• Peano-related dependencies 📦
• Output tensor construction 🧩

HOLD ON A SECOND LOLLLLL 🤣🤣🤣

Why do the problems just keep coming?!?!?!

Fix one thing, and another pops up!!

Fix that one, and HERE COMES THE NEXT ONE!!

Is this some kind of technical all-you-can-eat noodle challenge?!?! 🍜🍜🍜

Still, there's only one way forward: isolate each issue and tackle it one at a time!!

Header problems, initialization order, library dependencies, tensor construction...

We worked through them as separate issues and continued making adjustments!!

🧰 Separating the XRT Installation!! Let's Fix the Library Search Paths!!

One of the major steps in this investigation was this!!

Keep the system XRT intact, and install the SDK XRT separately!! 🔧✨

Here's what we prepared!!

• SDK XRT
• NPU-related components
• XDNA userspace plugin
• SDK ONNX Runtime
• Related dependencies such as Peano

Then, we configured the environment variables!!

XILINX_XRT LD_LIBRARY_PATH 

These were used to specify where the target process should look for its libraries!!

In other words, we kept the existing system environment while making the target program use the SDK-side libraries!! 😎

And here's the important part!!

Just putting the libraries somewhere isn't enough!!

We also need to know which libraries are being loaded, where they're being loaded from, and how their dependencies are resolved...

There's quite a bit to check here!! 🔍

By the way, "strace" also showed that the following library had been opened!!

/usr/lib/x86_64-linux-gnu/libxrt_coreutil.so.2 

Ooh!! We found a record showing that the library was opened!! 👀✨

...BUT DON'T JUMP TO CONCLUSIONS!!

We can't just say, "XRT initialization succeeded too!!"

Opening a library file and successfully initializing XRT are two different things!! ⚠️

We need to look carefully at what each log entry actually tells us!!

🎉 IT'S FINALLY HERE!! INFERENCE EXECUTION SUCCESS!!!!

And then...

AND THEN...!!!!

IT FINALLY HAPPENEDDDDDDD!!!! 🎊🎊🎊

Here's the execution log!!

Vitis AI EP registration: PASS ONNX session creation: PASS Running inference... Inference: PASS Quicktest: PASS 

WOOOOOOOOOOOOOOOOOO!!!! 🤣🔥🔥🔥

PASS!!!! PASS!!!! PASS!!!!

Execution Provider registration: SUCCESS!!

ONNX session creation: SUCCESS!!

And then...

🥳 Inference: PASS!!!!

WE DID ITTTTTTTTT!!!! 🎉💜🎉💜🎉

We've finally reached successful completion of the inference API execution!!!!

And there's more!!

We also got the following report about the NPU subgraph!!

Actually running on NPU 1 

OOOOOOOOH!!!! ⚡⚡⚡

We even got a report indicating that the NPU subgraph was actually executed on the NPU!!

That's a major step forward!! 👏✨

BUT!!!!

We still can't claim that the entire model ran exclusively on the NPU!!

The current results don't yet confirm the following!!

• Whether every operation in the model ran exclusively on the NPU
• Whether the output matches the CPU reference output numerically
• Whether the inference accuracy is correct
• How fast the actual inference is

Successful inference API execution? YES, that's confirmed!! But validating the entire model is still a task for later!!

Let's keep those two things separate!! ☝️

🔬 What Have We Learned? And What's Still Left to Do?

This investigation showed us that getting the NPU to work requires more than just hardware recognition!!

We need to get the entire software environment working together!!

Some of the key factors involved were...

• XRT versions and installation locations 🔧
• XDNA userspace plugin 🧩
• ONNX Runtime configuration 💻
• Dependencies such as Peano 📦
• Library search paths configured through environment variables 🔍
• Initialization and tensor construction procedures ⚙️

However, we haven't identified the XRT version difference as the sole cause of the issues!!

Since multiple components were changed, we still need to isolate which components affected which problems!!

The same goes for the memlock comparison!!

Some comparisons also involved changing the UID, so we can't conclude that memlock alone was responsible

And it doesn't mean that 1 GiB is the minimum required value, either!!

Also, the hashes of input archives and wheels collected later don't prove that they were identical to the inputs used during the earlier successful run!!

We'll need further testing to confirm reproducibility!! 🔍

🏃 What's Next?!

We're nowhere near done yet!!!!

Here's what we want to investigate next!!

① Numerical Comparison with CPU Reference Output 🔢

Compare the NPU output with the CPU reference output and check how much numerical difference there is!!

② Check Which Operations Actually Ran on the NPU 🧩

Find out which operations were assigned to the NPU and how much of the model actually ran there!!

③ Measure Inference Performance ⏱️

Let's find out how fast inference actually runs!!

④ Rebuild the Environment on Another Host 🖥️

Check whether we can reproduce the same setup on a different machine!!

⑤ Record the Successful Environment and Verify Reproducibility 📝

Keep proper records of input files, libraries, versions, and other details so we can reproduce the same results!!

WHOA, THERE'S STILL SO MUCH TO DO!! 🤣🤣🤣

But hey, we've made it this far, so now we can move on to these tests!!

🏁 Final Thoughts!! A Major Step Toward NPU Inference!!

In this investigation, we adjusted the XRT, Execution Provider, and related library configurations to use the NPU built into the Ryzen AI MAX+ 395 through ONNX Runtime!!

After encountering an ABI error with Python 3.14, we switched to using the ONNX Runtime C API from C++

We then installed the SDK XRT, related plugins, and dependencies separately, and used environment variables to specify the library search paths!!

And as a result...

WE FINALLY REACHED COMPLETION OF INFERENCE API EXECUTION!!!! 🎉🎉🎉

We also received a report indicating that the NPU subgraph was actually executed on the NPU!!

Of course, numerical comparisons, performance measurements, and other validations are still needed!!

Even so, this is a HUGE step forward!!

At first, XRT couldn't be found

Then Python threw an ABI error

Even after switching to C++, more problems appeared...

But by isolating and working through those issues one by one, we finally reached "Inference: PASS"!!

Simply recognizing the NPU isn't the finish line!! We've actually made it to the point where inference can run!!

That's a major milestone!! 🚀💜

All right!! Next up: numerical comparisons and performance measurements!!

The Ryzen AI MAX+ 395 NPU investigation is far from over!!!!

LET'S GOOOOOOO!!!! 🔥🔥🔥

Other editions / 他の言語・媒体

• 日本語 · Zenn
• English · Zenn
• English · DEV Community