I Build Insights, Not Features
A conversation with Martin Lonkwitz, R&D Software Engineer on the SAP on POWER team at IBM Germany, about compilers, kernel performance – and what sets research & development apart from conventional software development.
Dieses Gespräch gibt es auch auf Deutsch.
The SAP kernel is the heart of every SAP system: system software written in C/C++ that executes ABAP programs, manages processes and memory, and talks to the database. On IBM POWER systems running AIX and Linux, this kernel powers many of the largest data centers in the world. To keep it fast and stable there, a team at IBM Germany in Böblingen works exclusively for SAP. Martin Lonkwitz has been part of it since October 2020. We spoke with him.
webartikel.info: Mr. Lonkwitz, you describe your work as "research & development." To many people that sounds like a laboratory and a white coat. What do you actually do all day?
Martin Lonkwitz: No coat, but a great deal of measuring, yes. Our team makes sure the SAP kernel runs on AIX and Linux on POWER – and runs with the performance the platform can deliver. That includes support for the IBM XL C/C++ compiler SAP uses to build the kernel, shared responsibility for the kernel releases on AIX, and shared responsibility for the overall performance of every AIX kernel version.
In practice it works like this: when a customer somewhere in the world reports that their SAP system got slower after a kernel upgrade or a hardware migration, it lands with us. First we measure where the time is actually being lost – in which functions, in which loops. Then we read the affected source code, down to the machine instructions the compiler generated from it if necessary. And we talk to the people who build the tools: the compiler development team in Toronto and the AIX operating system developers. We propose fixes and improvements to them – and quite often our work results in official fixes for the IBM compiler and for AIX itself. At the end there is a root cause we can prove.
webartikel.info: At first that sounds like conventional development with debugging.
Lonkwitz: The difference is the question you ask. A developer on a product team asks: "How do I implement the feature that's in the backlog?" I ask: "What is even possible on this platform – and is it worth it?" I rarely build features. I build findings, prototypes and tools that eventually feed into a product – or don't, if it turns out the idea is no good. In R&D, a cleanly documented "no" is a result too.
webartikel.info: Let's take it step by step. A core R&D topic is technology evaluation. What does that look like for you?
Lonkwitz: For us, a great deal revolves around the compiler – the program that turns the developers' source code into the machine code the processor executes. With version 17, IBM moved the XL C/C++ compiler onto an LLVM/clang foundation, a generational shift compared with version 16. Every new version brings new optimization features, new code generation options, link-time optimization. My job is to check systematically: which of these measurably benefit the SAP kernel? What breaks? What behaves differently than documented? I build the kernel with the new options, run defined load scenarios, compare the measurements – and write down what I found.
It isn't only about the newest, though. We build all the way down to AIX 6.1 with the V11 compiler, because software SAP ships is still produced with it – the SAP Host Agent, for example. I personally hold primary responsibility for its AIX code: I implement new SAP monitoring functionality there and fix defects. So R&D doesn't mean leaving old platforms behind. It means having the whole range under control.
The same goes for programming languages. SAP also uses Rust for the SAP kernel on AIX, and we provide Rust support to SAP – to the same degree as for the C/C++ compiler versions. To that end I evaluated Rust in the low-level SAP/POWER environment by prototyping build, test and analysis tools in Rust. Not because Rust happens to be fashionable, but to have a solid statement about where it holds up on this platform and where it doesn't.
webartikel.info: And the hardware itself?
Lonkwitz: That too. A processor change – say from POWER9 to POWER10 – is never a non-event for a kernel tuned for performance. Cache behavior, the pipeline, the compiler's instruction selection: any of it can tip. You need to know that beforehand, not afterwards when the customer calls.
webartikel.info: Second point: prototyping and proofs of concept. The usual description says the code doesn't have to be perfect, it just has to demonstrate the idea. Do you see it that way?
Lonkwitz: Halfway. A PoC doesn't have to be production-ready – no error handling for every edge case, no end-user documentation, no internationalization. And that's as it should be, because a large share of our PoCs are answers to requirements: SAP brings questions to us, and we have to offer solutions on the AIX platform. One example is virtualization and orchestration – Kubernetes combined with versioned WPARs, AIX's container-like partitions. One of many requests. Then there are IBM-internal topics, such as AI workloads on POWER or automation with Ansible. In all of these cases the first question is feasibility, not a finished product.
But: the PoC has to be correct and reproducible. A prototype that shows a result nobody can reproduce isn't a feasibility study, it's an anecdote. For us a PoC is often a measurement setup: a kernel variant, a load profile, an evaluation. If I say "PGO delivers x percent on this workload," then the profile data, the build parameters and the raw measurements are sitting right next to that statement. Otherwise nobody would believe me, and rightly so – neither SAP nor the compiler people in Toronto.
webartikel.info: You mentioned profile-guided optimization. Can you explain that briefly for non-compiler-builders?
Lonkwitz: When compiling, a compiler has to guess which code paths run often and which rarely. PGO takes the guessing away: you build the kernel once in a measuring variant, run a representative workload, and collect profile data as it runs – which functions, which branches, how often. Then you compile a second time with that knowledge. The compiler places frequently used code together, optimizes the right loops, and saves function calls where it pays off.
The art isn't in pressing the button, it's in the measurement scenario. If the training profile doesn't reflect real customer operation, you optimize past the target. That's why I put a lot of time into the question of what a representative workload for an ABAP kernel even is – and how to measure the result so that it holds up to scrutiny.
webartikel.info: Third point: basic research – algorithms for which no standard solution exists yet. Does that happen in your field?
Lonkwitz: In a more modest sense than in AI research, yes. There is no standard solution for how to optimally compile a kernel that has grown over decades for a specific CPU generation. There is no ready-made method for tracing a regression that only appears at one customer under one specific load back to a single compiler decision. You work that out yourself: with measurement tools, with your own analysis programs, sometimes with a look at the generated machine code. To me, fundamentals means understanding what actually happens beneath the surface.
That's not to say experience plays no role – quite the opposite. Knowledge can be delegated; understanding cannot. And understanding includes good intuition, which grows out of experience, plus a healthy dose of creativity. That, incidentally, is exactly the capability that sits above what an AI can contribute today: it has the knowledge, but the intuition about where to look comes from the person who has watched these systems under load for decades. Anyone who sees only dry numbers in scientific work is taking a lot of potential away from themselves.
webartikel.info: Fourth point: patents and white papers. Do you write scientific papers?
Lonkwitz: No conference papers in the academic sense. But I write a great deal – root cause analyses, defect reports, evaluation reports. That's the equivalent. A defect report to a compiler development team has to be structured like a small scientific paper: environment, symptom, evidence, defect statement, reproduction, workarounds, requested action, attached evidence. My rule: headings are promises. If a section is titled "Root Cause," what follows is the cause – proven – and not a hypothesis. And no claim without a reference to data. That sounds strict, but it saves everyone involved weeks.
I used to give technical talks, for instance in 2009 at the Vienna Technology Congress on performance in Fibre Channel networks. Today, knowledge sharing for me happens more in onboarding new colleagues. When I joined this team, it was still the old guard – people who grew up at IBM and had worked in this field for several decades. There was an enormous amount to learn, and I think I used those first years well. Today I try to pass that on.
webartikel.info: Fifth point: knowledge transfer. What happens to your results?
Lonkwitz: That's the whole point. R&D that doesn't end up in a product is a hobby. For us, transfer runs in two directions. Upstream, to the compiler and the operating system: when we show that an optimization produces incorrect or slow code, or that AIX doesn't behave the way it should at some point, it goes to the development team as a defect and gets fixed there – and every XL C/C++ and AIX user benefits, not just SAP. And toward the product: when a compiler option or a PGO setup measurably delivers, it goes to SAP with a recommendation to adopt it in the official kernel build. That's where the colleagues sit who turn it into a stable release, with everything that entails.
My work ends with a proven result and a handover. You have to be able to live with that: you build things that other people finish.
webartikel.info: But you also have conventional development responsibilities, correct?
Lonkwitz: Yes, for us it's not either-or. I'm directly responsible for several kernel components: the SAP monitoring saposmon, the CIM code, the AIX part of the SAP Host Agent, and the Google Robot test framework. saposmon is the piece of the kernel that tells the SAP system how the operating system is doing – CPU, memory, paging, utilization. On AIX with LPARs, micro-partitioning and shared processors, "how much CPU do I have right now?" is a surprisingly deep question. If saposmon delivers wrong values there, administrators make wrong decisions. That's operational responsibility in the classic sense: fixing bugs, maintaining releases, keeping tests green. Add to that AIX-specific solutions, such as Ansible-based automation and RBAC configurations.
I also support the colleague on our team who directly handles SAP customer tickets around AIX and Linux on POWER. When a case there goes deeper than a configuration problem, it comes to me – and conversely, those tickets show me what is actually happening out at the customers.
I think the mix is healthy. Those who only research lose the feel for what really hurts in operations. Those who only operate stop asking questions.
webartikel.info: How did you end up in this niche? Your career path is anything but straight.
Lonkwitz: It only looks crooked if you go by job titles. I started in the German Armed Forces as an IT specialist – UNIX networks, military simulation, deployed abroad in 1997. Then several years of Unix engineering for banks and industry, in Frankfurt, Neuss and Tokyo, on large SUN systems. From 2002 on, Lufthansa Systems: first Linux and z/VM on the IBM mainframe, back when that was new, then ten years of SAN and storage operations for Lufthansa and its customers, with almost four thousand Fibre Channel ports. I wrote my own monitoring and automation tools there in C and Perl because nothing suitable existed. Later, product development on the mainframe: zGuard, a capacity and resource control system in z/Assembler and C++ that ran at insurers, banks and in the automotive industry.
The common thread is always the same: system software close to the hardware, in C and C++, for infrastructure that must not fail. The SAP kernel on POWER is simply the most consistent form of that.
webartikel.info: What should someone bring along who wants to move toward R&D in the systems field?
Lonkwitz: Three things. First, patience with measurements – most of the time doesn't go into programming, it goes into building clean comparisons. Second, the willingness to go beneath the abstraction: read compiler output, understand operating system behavior, read hardware documentation. And third, honesty toward your own result. The biggest mistake in R&D is reporting the result you wished for. An idea cleanly disproven after two weeks is a good outcome. An idea you talk up for three years is an expensive one.
webartikel.info: Mr. Lonkwitz, thank you for the conversation.
Martin Lonkwitz has worked since 2020 as an R&D Software Engineer on the SAP on POWER team at IBM Germany in Böblingen. Focus areas: SAP kernel development on AIX and Linux on POWER, IBM XL C/C++ compiler and Rust support, profile-guided optimization, SAP monitoring (saposmon, CIM, SAP Host Agent).