Programmatically Toggle Specific LLVM Optimizations

Hi everyone!

First time poster here. Apologies if I am breaking some rules and please let me know so I will not break it again!

I am a summer research student at the Computer Science Department at the University of Toronto and I am working on benchmarking LLVM optimizations.

I tried looking for it online but was not able to find a programmatic way to toggle specific LLVM optimizations. For example, I would like to compile using the -O3 flag but with LICM turned off to see how much the compiled binary degrades in performance. I would like the ability to simultaneously toggle a few optimizations together as well. I would like to benchmark many optimizations so I would like to program the experiments.

So to be really specific, there are two questions:

1. Are there some command line interfaces that can accomplish this task? Is there a way to write a bash script to enumerate through the optimizations and turning them off one a time?

2. If not, what is a good guide to PassManager so that I can program this functionality?

Thanks so much for your help!

Sincerely,
Qiongsi

UTEA Reserach Student 2018, EcoSystem
Computer Science, University of Toronto

Some passes have supported options to disable them, e.g.
-fno-vectorize and -fno-unroll-loops, but there is no general option.
Since it's not useful in general to disable arbitrary options, some
handywork is required. Using the legacy (=default) pass manager here.

To get the passes that are run:
$ clang -O3 -mllvm -debug-pass=Arguments

To optimize separately:
$ clang -Xclang -disable-llvm-passes -S -emit-llvm -o - | opt <your
list of passes> | llc
(not tested, so don't expect to work straight out of the box)

This is not strictly the same as optimizing using clang (pass options
are missing), so there will be performance differences. You could
instead customize PassManagerBuilder::populateModulePassManager() and
PassManagerBuilder::populateFunctionPassManager().

Michael

Hi Michael!

Thanks for your reply. I managed to add a small amount of code to PassManager to have this feature implemented.

One can now turn off a pass by doing

opt -disablepass=licm or clang -mllvm -disablepass=licm

to turn off, for example, loop invariant code motion.

I programmed the command line option to be really hidden. This maybe a useful thing to have in the trunk for people who want to fine tune the optimizations.

Should I submit a patch? I am not sure if my code follows all the best practices of the community, but it is working from my own testing.

Thanks again for your help!

Qiongsi

Unless you intend to upstream you change (which I encourage you to),
there is no need to follow any best practices. If you intend to submit
the patch, the code reviews exist to ensure that the best practices
are followed. You can avoid the most obvious change requests by
considering LLVM Coding Standards — LLVM 16.0.0git documentation .

Please note that there is a newer pass manager. You might get feedback
to update the that pass manager as well/instead or that your use case
is too 'niche' to be useful for others. However, you won't find out
before submitting the patch. And the experience might be useful if you
intend collaborate to LLVM more often.

Michael

Thanks for the fast reply! I will submit a patch and find out!

Sincerely,
Qiongsi

Could you please share what change you make to disable a pass .
I am also trying to disable a specific pass .

Here is what I did. This was really old and only worked with the legacy pass manager.

From e107781150eaef0eecdca5437debd49c360b9db8 Mon Sep 17 00:00:00 2001
From: Qiongsi Wu <qiongsiwu@gmail.com>
Date: Sat, 23 Jun 2018 16:55:56 -0400
Subject: [PATCH 1/3] Adding cmdline option to skip passes

---
 tools/opt/opt.cpp | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/tools/opt/opt.cpp b/tools/opt/opt.cpp
index 43771d5d75bb..7d1f4f73d439 100644
--- a/tools/opt/opt.cpp
+++ b/tools/opt/opt.cpp
@@ -265,17 +265,42 @@ static cl::opt<std::string>
                     cl::desc("YAML output filename for pass remarks"),
                     cl::value_desc("filename"));
 
+/****************************************************************************/
+// List of passes to disable
+// Use syntax
+// -disablepass pass_arg1 -disablepass pass_arg2
+// to disable multiple passes at the same time
+static cl::list<std::string> DisabledPasses("disablepass",
+                                           cl::desc("Disable the specified pass"),
+                                           cl::value_desc("passname"),
+                                           cl::ReallyHidden);
+/****************************************************************************/
+
 class OptCustomPassManager : public legacy::PassManager {
 public:
   using super = legacy::PassManager;
 
   void add(Pass *P) override {
+    AnalysisID AID = P->getPassID();
+    const PassInfo *PInfo = PassRegistry::getPassRegistry()->getPassInfo(AID);
+    StringRef PArg = PInfo->getPassArgument();
+
+    // Hack to skip passes if disabled
+    for (unsigned i = 0; i < DisabledPasses.size(); i++) {
+        if (PArg == DisabledPasses[i]) {
+            dbgs() << "Pass disabled: "
+                   << PArg << "\n";
+            return;
+        }
+    }
+      
     bool WrapWithDebugify = DebugifyEach && !P->getAsImmutablePass() &&
                             !isIRPrintingPass(P) && !isBitcodeWriterPass(P);
     if (!WrapWithDebugify) {
       super::add(P);
       return;
     }
+
     PassKind Kind = P->getPassKind();
     // TODO: Implement Debugify for BasicBlockPass, LoopPass.
     switch (Kind) {

From 39c57d6fa7fe745edf4082b919375375f1b23cb8 Mon Sep 17 00:00:00 2001
From: Qiongsi Wu <qiongsiwu@gmail.com>
Date: Sat, 23 Jun 2018 17:39:43 -0400
Subject: [PATCH 2/3] Adding code to disable optimizations in
 FunctionPassManager

---
 tools/opt/opt.cpp | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/tools/opt/opt.cpp b/tools/opt/opt.cpp
index 7d1f4f73d439..0254b68a03d0 100644
--- a/tools/opt/opt.cpp
+++ b/tools/opt/opt.cpp
@@ -279,7 +279,7 @@ static cl::list<std::string> DisabledPasses("disablepass",
 class OptCustomPassManager : public legacy::PassManager {
 public:
   using super = legacy::PassManager;
-
+  
   void add(Pass *P) override {
     AnalysisID AID = P->getPassID();
     const PassInfo *PInfo = PassRegistry::getPassRegistry()->getPassInfo(AID);
@@ -321,6 +321,30 @@ class OptCustomPassManager : public legacy::PassManager {
   }
 };
 
+class OptDisableFunctionPassManager : public legacy::FunctionPassManager {
+public:
+    using super = legacy::FunctionPassManager;
+    
+    explicit OptDisableFunctionPassManager(Module *M) : super(M) {}
+    
+    void add(Pass *P) override {
+        AnalysisID AID = P->getPassID();
+        const PassInfo *PInfo = PassRegistry::getPassRegistry()->getPassInfo(AID);
+        StringRef PArg = PInfo->getPassArgument();
+        
+        // Hack to skip passes if disabled
+        for (unsigned i = 0; i < DisabledPasses.size(); i++) {
+            if (PArg == DisabledPasses[i]) {
+                dbgs() << "Pass disabled: "
+                << PArg << "\n";
+                return;
+            }
+        }
+        
+        super::add(P);
+    }
+};
+
 static inline void addPass(legacy::PassManagerBase &PM, Pass *P) {
   // Add the pass to the pass manager...
   PM.add(P);
@@ -651,7 +675,8 @@ int main(int argc, char **argv) {
   std::unique_ptr<legacy::FunctionPassManager> FPasses;
   if (OptLevelO0 || OptLevelO1 || OptLevelO2 || OptLevelOs || OptLevelOz ||
       OptLevelO3) {
-    FPasses.reset(new legacy::FunctionPassManager(M.get()));
+    //FPasses.reset(new legacy::FunctionPassManager(M.get()));
+    FPasses.reset(new OptDisableFunctionPassManager(M.get()));
     FPasses->add(createTargetTransformInfoWrapperPass(
         TM ? TM->getTargetIRAnalysis() : TargetIRAnalysis()));
   }

From 8d1c6bd221631e9b56fa9cede80d80646f8b4aec Mon Sep 17 00:00:00 2001
From: Qiongsi Wu <qiongsiwu@gmail.com>
Date: Tue, 26 Jun 2018 17:12:30 -0400
Subject: [PATCH 3/3] Changing code so that the disablepass option works for
 both clang and opt.

---
 include/llvm/IR/LegacyPassManager.h |  9 +++++
 lib/IR/LegacyPassManager.cpp        | 55 +++++++++++++++++++++++++++++
 tools/opt/opt.cpp                   | 50 +-------------------------
 3 files changed, 65 insertions(+), 49 deletions(-)

diff --git a/include/llvm/IR/LegacyPassManager.h b/include/llvm/IR/LegacyPassManager.h
index 9a376a151505..d1e18463c5c7 100644
--- a/include/llvm/IR/LegacyPassManager.h
+++ b/include/llvm/IR/LegacyPassManager.h
@@ -92,6 +92,15 @@ class FunctionPassManager : public PassManagerBase {
   FunctionPassManagerImpl *FPM;
   Module *M;
 };
+    
+class OptDisableFunctionPassManager : public legacy::FunctionPassManager {
+public:
+    using super = legacy::FunctionPassManager;
+    
+    explicit OptDisableFunctionPassManager(Module *M) : super(M) {}
+    
+    void add(Pass *P) override;
+};
 
 } // End legacy namespace
 
diff --git a/lib/IR/LegacyPassManager.cpp b/lib/IR/LegacyPassManager.cpp
index b04787cb30eb..23f6eb36d95e 100644
--- a/lib/IR/LegacyPassManager.cpp
+++ b/lib/IR/LegacyPassManager.cpp
@@ -60,6 +60,17 @@ PassDebugging("debug-pass", cl::Hidden,
   clEnumVal(Executions, "print pass name before it is executed"),
   clEnumVal(Details   , "print pass details when it is executed")));
 
+/****************************************************************************/
+// List of passes to disable
+// Use syntax
+// -disablepass=pass_arg1 -disablepass=pass_arg2
+// to disable multiple passes at the same time
+static cl::list<std::string> DisabledPasses("disablepass",
+                                            cl::desc("Disable the specified pass"),
+                                            cl::value_desc("passname"),
+                                            cl::ReallyHidden);
+/****************************************************************************/
+
 namespace {
 typedef llvm::cl::list<const llvm::PassInfo *, bool, PassNameParser>
 PassOptionList;
@@ -96,6 +107,38 @@ static cl::list<std::string>
                             "options"),
                    cl::CommaSeparated, cl::Hidden);
 
+/****************************************************************************/
+// Static function to determine if a pass should be skipped
+
+static bool SkipPass(Pass *P) {
+    AnalysisID AID = P->getPassID();
+    const PassInfo *PInfo = PassRegistry::getPassRegistry()->getPassInfo(AID);
+    
+    // There are passes that are somehow not registered
+    // but are added. For example
+    // SpeculativeExecutionLegacyPass
+    // Such passes are not skipped because if they are not registered,
+    // opt should not be able to use it, and we don't have to skip it.
+    // Need suggestions on how to do this better.
+    if (!PInfo) {
+        return false;
+    }
+    
+    StringRef PArg = PInfo->getPassArgument();
+    
+    for (unsigned i = 0; i < DisabledPasses.size(); i++) {
+        if (PArg == DisabledPasses[i]) {
+            dbgs() << "Pass disabled: "
+                   << PArg << "\n";
+            return true;
+        }
+    }
+    
+    return false;
+}
+
+/****************************************************************************/
+
 /// This is a helper to determine whether to print IR before or
 /// after a pass.
 
@@ -1467,6 +1510,16 @@ bool FunctionPassManager::doFinalization() {
   return FPM->doFinalization(*M);
 }
 
+//===----------------------------------------------------------------------===//
+// OptDisableFunctionPassManager
+
+void OptDisableFunctionPassManager::add(Pass *P) {
+    if (SkipPass(P))
+        return;
+    super::add(P);
+}
+
+
 //===----------------------------------------------------------------------===//
 // FunctionPassManagerImpl implementation
 //
@@ -1786,6 +1839,8 @@ PassManager::~PassManager() {
 }
 
 void PassManager::add(Pass *P) {
+  if (SkipPass(P))
+      return;
   PM->add(P);
 }
 
diff --git a/tools/opt/opt.cpp b/tools/opt/opt.cpp
index 0254b68a03d0..7b21245d9518 100644
--- a/tools/opt/opt.cpp
+++ b/tools/opt/opt.cpp
@@ -265,35 +265,11 @@ static cl::opt<std::string>
                     cl::desc("YAML output filename for pass remarks"),
                     cl::value_desc("filename"));
 
-/****************************************************************************/
-// List of passes to disable
-// Use syntax
-// -disablepass pass_arg1 -disablepass pass_arg2
-// to disable multiple passes at the same time
-static cl::list<std::string> DisabledPasses("disablepass",
-                                           cl::desc("Disable the specified pass"),
-                                           cl::value_desc("passname"),
-                                           cl::ReallyHidden);
-/****************************************************************************/
-
 class OptCustomPassManager : public legacy::PassManager {
 public:
   using super = legacy::PassManager;
   
   void add(Pass *P) override {
-    AnalysisID AID = P->getPassID();
-    const PassInfo *PInfo = PassRegistry::getPassRegistry()->getPassInfo(AID);
-    StringRef PArg = PInfo->getPassArgument();
-
-    // Hack to skip passes if disabled
-    for (unsigned i = 0; i < DisabledPasses.size(); i++) {
-        if (PArg == DisabledPasses[i]) {
-            dbgs() << "Pass disabled: "
-                   << PArg << "\n";
-            return;
-        }
-    }
-      
     bool WrapWithDebugify = DebugifyEach && !P->getAsImmutablePass() &&
                             !isIRPrintingPass(P) && !isBitcodeWriterPass(P);
     if (!WrapWithDebugify) {
@@ -321,30 +297,6 @@ class OptCustomPassManager : public legacy::PassManager {
   }
 };
 
-class OptDisableFunctionPassManager : public legacy::FunctionPassManager {
-public:
-    using super = legacy::FunctionPassManager;
-    
-    explicit OptDisableFunctionPassManager(Module *M) : super(M) {}
-    
-    void add(Pass *P) override {
-        AnalysisID AID = P->getPassID();
-        const PassInfo *PInfo = PassRegistry::getPassRegistry()->getPassInfo(AID);
-        StringRef PArg = PInfo->getPassArgument();
-        
-        // Hack to skip passes if disabled
-        for (unsigned i = 0; i < DisabledPasses.size(); i++) {
-            if (PArg == DisabledPasses[i]) {
-                dbgs() << "Pass disabled: "
-                << PArg << "\n";
-                return;
-            }
-        }
-        
-        super::add(P);
-    }
-};
-
 static inline void addPass(legacy::PassManagerBase &PM, Pass *P) {
   // Add the pass to the pass manager...
   PM.add(P);
@@ -676,7 +628,7 @@ int main(int argc, char **argv) {
   if (OptLevelO0 || OptLevelO1 || OptLevelO2 || OptLevelOs || OptLevelOz ||
       OptLevelO3) {
     //FPasses.reset(new legacy::FunctionPassManager(M.get()));
-    FPasses.reset(new OptDisableFunctionPassManager(M.get()));
+      FPasses.reset(new legacy::OptDisableFunctionPassManager(M.get()));
     FPasses->add(createTargetTransformInfoWrapperPass(
         TM ? TM->getTargetIRAnalysis() : TargetIRAnalysis()));
   }

Thanks for your reply. I want this disable-pass to implement in the new pass manager and also my requirement is to disable O3 level passes one by one to check the performance.

Please let me know if you have any ideas to implement this.

Thanks
Soma

In another thread, @mshockwave mentioned PassInstrumentation, I think, which could maybe selectively disable passes.

I implemented disable flag to disable specific passes .

Thanks a lot for helping .