How Red Hat came to create the OpenJDK AArch64 port for 64-bit Arm
It’s been quite a year for Arm. At the end of 2019, Amazon Web Services announced Graviton2 servers, which have Arm-based rather than Intel x86 processors. Then in June 2020, Apple announced that they’d be moving Macintosh computers over to use Apple silicon, which means Arm. I’ve been expecting this to happen for a very long time. In truth, I’d almost given up: time and again there’d be an announcement about some Arm-based servers or desktop machine, but then it’d fizzle out like a damp firework, and nothing would happen.
I’ve been following the fortunes of Arm-based servers for a long time. Wind back to a day in 2011, and I had lunch at The Wrestlers (the Cambridge UK tech community’s favourite Thai eatery) with Jon Masters, the lead Arm Architect at Red Hat. He told me some exciting news: Arm were finally going to produce a 64-bit architecture, and Red Hat was going to port Enterprise Linux to it. There were a lot of problems to solve in order to get this done, but one was particularly difficult: there was no Java. Java is a key component of much Enterprise software, so this was a pretty big deal. I’d been involved in the OpenJDK project for some time at this point, so I was the obvious person to ask. The choice was pretty simple: we’d have either to write it ourselves or we’d have to pay someone else to do it. I had heard that a bare-bones port could be done by two experts in about a year, but that was all. If my team was going to do it, we’d have to learn as we went along.
I was thrilled at the prospect of getting my teeth into a really substantial piece of work, but I had a problem: how on Earth was I going to sell the idea of doing such a substantial project to Red Hat management, with no guarantee of success? I reasoned that unless we did the port ourselves we’d have to pay Oracle to do it, and that would cost lots, so why not save money by doing it ourselves? It’d be good publicity, and we’d be able to build alliances. Thankfully our management agreed to go along with this, so we were good to go.
Everything you need to grow your career.
With your free Red Hat Developer program membership, unlock our library of cheat sheets and ebooks on next-generation application development.
SIGN UP
I was determined that I’d be one of the engineers doing the work, but I needed someone else. Fortunately, Andrew Dinn was about to join our Java team. He’d worked on Java for a long time, but perhaps more importantly had some experience on compilers for Lisp machines and logic languages, so he’d be a good fit. One thing that worried me, though: what if someone else did a port? In particular, what if Oracle did a port? Then the whole effort might go to waste, along with any hope of glory. The only way to prevent that disaster, I figured, was to gain first-mover advantage. Get it done fast, do it well, and make it free. Build it, and they will come.
There was, however, a problem, or rather two problems. The processor did not exist, and wasn’t going to exist for some time. Even worse, persuading Arm to give us the detailed information we needed about the architecture was not at all easy. Some persistence and the help of Arm’s Philippe Robin solved the documentation problem, but the lack of hardware was more difficult. It is possible to run all of OpenJDK under simulation, but the simulators at the time were painfully slow, impractical for Java. Andrew Dinn remembers being at breakfast at a technical conference, where I excitedly told him I’d figured out what to do while in the shower! We’d use an instruction set simulator, but just for the AArch64 code that we generated ourselves. The rest of the HotSpot VM is written in C++, which we could run natively at full speed on an Intel x86-based PC. Essentially, the AArch64 simulator would be a library we’d link into the JVM. But where would we get an AArch64 simulator library? “We’ll write one,” I said. “It’s a RISC. How hard can it be?”
We started in the June of 2012. It took Andrew Dinn a little while to write the simulator while I got on with writing the assembler and the initial startup code, but after a couple of months we were executing Java bytecodes. The decision to write our own simulator turned out to have been inspired: we could create complex breakpoint conditions and even record instruction traces so that when HotSpot crashed we could see the instruction history. This made the startup process go really quickly, and the logs reveal the commit on Oct 4 2012: “Enough for Hello, World!” This was an important day: in order to get as far as outputting “Hello, World!” to the console, Java has to execute about three-quarters of a million bytecodes without crashing, and this exercises much of the virtual machine.
By June 2013, there were three of us: Edward Nevill, a Java expert formerly of Arm, joined the project from Linaro, and he was first to get actual hardware. I was so comfortable with our little AArch64 simulator that I was very happy for someone else to debug our work on the real machine, which I expected to be a painful process. But then came a revelation: apart from a small problem with flushing the instruction cache, it worked! I was astonished: we’d written our own simulator, compiler, and assembler from Arm’s documentation, and we had no independent verification that we’d done it right. Edward was a huge help, and by the end of 2013 we had a working port. Hat began shipping early releases to its partners, we carried on fixing bugs and improving performance, and on March 2, 2015, the AArch64 port was integrated into the main OpenJDK project.
But there’s a huge amount of difference between a working port and one that’s really good. Every efficient system is composed of thousands of tiny optimizations, each one almost insignificant on its own. I don’t know how much time has been spent on the Intel/x86 port, but it must be dozens of engineer-years, and we really needed to make the AArch64 port competitive. We needn’t have worried. Over the next few years interest grew, some people from the AArch64 silicon producers and other companies joined us, and now there are many people working on AArch64 Java from all round the world. This is where Red Hat’s open and collaborative approach really pays off: by forming partnerships with others, we gain a big multiplier over purely in-house software development.
But that’s not the end of the story. I’d heard that Oracle were running on 64-bit Arm, and I wondered if it was our AArch64 port they were running. Oracle hadn’t said anything about it, but maybe they were. It turned out that it wasn’t our port, it was something of their own, based on the (32-bit) Arm port they already had. At this point I really started to get worried they’d get much better performance than us, and all of our work would be wasted: after all, they were the experts, the originators of Java, and we were new to it. Eventually I found someone who had used Oracle’s port, and they kindly assured me that I had nothing to worry about. It still seems strange to me that this happened: Oracle had permission from the very beginning to use all of our code, so why write it again? I still don’t really know, but there were apparently some commercial considerations.
Remember that Oracle port? On November 12, 2020, Oracle announced that they’d decided to focus their resources going forward on a single 64-bit Arm port, and that it was the AArch64 port we’d started. Not only that, but the 64-bit Arm support in their maintenance release of JDK 8 is now based on our port as well. So why did this happen? I believe it’s because we stepped up with a working port early, got it into the OpenJDK mainline, and welcomed everyone who wanted to participate. We kept as many of our discussions as possible open, people volunteered, and we built a community. We achieved far more with this community of developers that we ever could hope to have done as two individuals, but none of it would have happened unless we’d released that complete working port back in 2014.
