如何重新打包HttpClient 4.3.1并删除对commons-logging的依赖？

Question

我想重新打包apache的httpclient lib,用一个Android应用程序发送它(比如https://code.google.com/p/httpclientandroidlib/但是使用HttpClient 4.3.1)

因此,我手动下载了httpclient 4.3.1 jar(包括其所有依赖项)并使用jarjar重新打包它:

x@x$: cd libs && for f in *.jar; do java -jar ../jarjar-1.4.jar process ../rules.txt $f out/my-$f; done

使用rules.txt:

 rule org.apache.http.** my.repackaged.org.apache.http.@1

然后我用ant将输出放在一起:

<project name="MyProject" default="merge" basedir=".">
  <target name="merge">
        <zip destfile="my-org-apache-httpclient-4.3.1.jar">
            <zipgroupfileset dir="libs/out" includes="*.jar"/>
        </zip>
  </target>
</project>

我可以使用该文件来开发和测试我的应用程序,但如果我在android上部署它,它会抛出一个异常,就像它找不到my.repackaged.org.apache.logging.log4j.something引用的那样my.package.org.apache.logging.whatEver.

所以,现在我想通过使用字节码操作来消除对commons-logging的任何依赖.这已经在以前完成:http://sixlegs.com/blog/java/dependency-killer.html

但我想知道我是怎么做到的？org.apache.commons.logging.Log只有依赖项:

x$x$: java -jar jarjar-1.4.jar find jar my-org-apache-httpclient-4.3.1.jar commons-logging-1.1.3.jar
my/http/impl/execchain/ServiceUnavailableRetryExec -> org/apache/commons/logging/Log
my/http/impl/execchain/RetryExec -> org/apache/commons/logging/Log
my/http/impl/execchain/RedirectExec -> org/apache/commons/logging/Log
my/http/impl/execchain/ProtocolExec -> org/apache/commons/logging/Log
...

我认为要走的路是,删除这些依赖项并将其替换为自己的实现,就像他在此处所做的那样https://code.google.com/p/httpclientandroidlib/.因此,我创建了一个新的maven项目,只有一个类具有providedcommons-logging的作用域,它实现了org.apache.commons.logging.Log接口,只是删除了android.utils.Log:

MyLog implements org.apache.commons.logging.Log {}

在包中my.log,我将其打包在my-log-1.0.0.jar中.我将该jar放入与重新打包的httpclient-jars相同的文件夹中,并使用上面提到的ant将所有内容打包在my-org-apache-httpclient-4.3.1.jar中.

---

方法1

我再次尝试使用jarjar:

java -jar jarjar-1.4.jar process rules2.txt my-org-apache-httpclient-4.3.1.jar my-org-apache-httpclient-4.3.1-without-logging-dep.jar

与rules2.txt:

rule my.repackaged.commons.logging.** my.log.@1

但这不起作用.仍然抛出了它无法找到my.repackaged.org.apache.logging.log4j.something引用的my.package.org.apache.logging.whatEver异常.

---

方法2

我还尝试从最终jar中删除日志记录内容和/或重新打包my.repackaged.org.apache.log4j并记录到其原始包:

rules2.txt v2:

rule my.repackaged.org.apache.log4j.** org.apache.log4j.@1
rule my.repackaged.org.apache.logging.** org.apache.logging.@1

但这仍然是抛出的赘述:my.repackaged.org.apache.logging.log4j.something引用my.package.org.apache.logging.whatEver

---

题

如何杀死/替换commons-logging依赖项并摆脱异常？

Answer 1

介绍

如果程序依赖于库,则通常意味着它使用库的方法.因此,删除依赖关系不是一项简单的任务.您实际上想要删除程序所需的代码 - 至少是正式的 - .

删除依赖项有三种方法:

1. 调整源代码以使其不依赖于库并从头开始编译.
2. 修改字节码以删除对项目所依赖的库的引用.
3. 操作运行时不需要依赖项.最简单的方法是重新创建所需的类并将它们放入jar文件中.

这些方式都不是很漂亮.所有这些都需要大量的工作.没有保证没有副作用.

解

我将通过介绍用于解决问题的文件和步骤来描述我的解决方案.要重现,您将需要以下文件(在单个目录中):

lib/xxx-vvvjar:库jars(httpclient和依赖项,不包括 commons-logging-1.1.3.jar)
jarjar-1.4.jar:用于重新打包jar 规则
.txt:jarjar规则

rule org.apache.http.** my.http.@1
rule org.apache.commons.logging.** my.logging.@1

build.xml:Ant构建配置

<project name="MyProject" basedir=".">
    <target name="logimpl">
        <javac srcdir="java/src" destdir="java/bin" target="1.5" />
        <jar jarfile="out/logimpl.jar" basedir="java/bin" />
    </target>
    <target name="merge">
        <zip destfile="httpclient-4.3.1.jar">
            <zipgroupfileset dir="out" includes="*.jar"/>
        </zip>
    </target>
</project>

的java/SRC/Log.java

package my.logging;

public interface Log {
    public boolean isDebugEnabled();
    public void debug(Object message);
    public void debug(Object message, Throwable t);

    public boolean isInfoEnabled();
    public void info(Object message);
    public void info(Object message, Throwable t);

    public boolean isWarnEnabled();
    public void warn(Object message);
    public void warn(Object message, Throwable t);

    public boolean isErrorEnabled();
    public void error(Object message);
    public void error(Object message, Throwable t);

    public boolean isFatalEnabled();
    public void fatal(Object message);
    public void fatal(Object message, Throwable t);
}

的java/SRC/LogFactory.java

package my.logging;

public class LogFactory {

    private static Log log;

    public static Log getLog(Class<?> clazz) {
        return getLog(clazz.getName());
    }

    public static Log getLog(String name) {
        if(log == null) {
            log = new Log() {
                public boolean isWarnEnabled() { return false; }
                public boolean isInfoEnabled() { return false; }
                public boolean isFatalEnabled() { return false; }
                public boolean isErrorEnabled() {return false; }
                public boolean isDebugEnabled() { return false; }
                public void warn(Object message, Throwable t) {}
                public void warn(Object message) {}
                public void info(Object message, Throwable t) {}
                public void info(Object message) {}
                public void fatal(Object message, Throwable t) {}
                public void fatal(Object message) {}
                public void error(Object message, Throwable t) {}
                public void error(Object message) {}
                public void debug(Object message, Throwable t) {}
                public void debug(Object message) {}
            };
        }
        return log;
    }

}

do_everything.sh

#!/bin/sh

# Repackage library
mkdir -p out
for jf in lib/*.jar; do
    java -jar jarjar-1.4.jar process rules.txt $jf `echo $jf | sed 's/lib\//out\//'`
done

# Compile logging implementation
mkdir -p java/bin
ant logimpl

# Merge jar files
ant merge

而已.打开一个控制台并执行

cd my_directory && ./do_everything.sh

这将创建一个包含单个jar文件的文件夹"out"和"httpclient-4.3.1.jar",它是最终的,独立且工作的jar文件.那么,我们刚刚做了什么？

1. 重新打包的httpclient(现在my.http)
2. 修改了库my.logging而不是使用org.apache.commons.logging
3. 库(my.logging.Log和my.logging.LogFactory)使用的编译所需类.
4. 将重新打包的库和已编译的类合并到一个jar文件httpclient-4.3.1.jar中.

很简单,不是吗？只需逐行阅读shell脚本即可发现单个步骤.要检查是否已删除所有依赖项,您可以运行

java -jar jarjar-1.4.jar find class httpclient-4.3.1.jar commons-logging-1.1.3.jar

我尝试使用SE7和Android 4.4生成的jar文件,它在两种情况下都有效(请参阅下面的备注).

类文件版本

每个类文件都有一个主要版本和一个次要版本(都取决于编译器).Android SDK要求类文件的主要版本小于0x33(所以1.7/JDK 7之前的所有版本).我将该target="1.5"属性添加到ant javac任务中,因此生成的类文件的主要版本为0x31,因此可以包含在您的Android应用程序中.

替代(字节码操作)

你很幸运 记录(几乎总是)是单向操作.它几乎不会引起影响主程序的副作用.这意味着应该可以删除commons-logging,因为它不会影响程序的功能.

我选择了你在问题中建议的第二种方式,字节码操作.这个概念基本上就是这个(A是httpclient,B是commons-logging):

1. 如果A的方法的返回类型是B的一部分,则返回类型将更改为java.lang.Object.
2. 如果A的方法的任何参数具有属于B的类型,则参数类型将更改为java.lang.Object.
3. 完全删除属于B的方法的调用.pop并插入常量指令以修复VM堆栈.
4. 从A调用的方法的描述符中删除属于B的类型.这需要处理目标类(包含被调用方法的类).属于B的所有对象类型都将替换为java.lang.Object.
5. 将删除尝试访问属于B的类字段的指令.pop并插入常量指令以修复VM堆栈.
6. 如果方法尝试访问属于B的类型的字段,则指令引用的字段签名将更改为java.lang.Object.这需要处理目标类(包含访问字段的类).
7. 包含在B中但属于A类的字段被修改,以便它们的类型为java.lang.Object.

正如您所看到的,这背后的想法是用java.lang.Object和替换所有引用的类,并删除对属于commons-logging的类成员的所有访问.

我不知道这是否可靠,并且在应用操纵器后我没有测试库.但是从我所看到的(反汇编的类文件和加载类文件时没有VM错误)我相当确定代码是有效的.

我试图记录程序的几乎所有内容.它使用ASM Tree API,它提供了对类文件结构的相当简单的访问.并且 - 为了避免不必要的负面评论 - 这是"快速'脏'代码.我没有真正测试它,我打赌有更快的字节码操作方式.但是这个程序似乎满足了OP的需求,这就是我所写的.

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.List;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.jar.JarOutputStream;

import org.objectweb.asm.ClassReader;
import org.objectweb.asm.ClassWriter;
import org.objectweb.asm.Opcodes;
import org.objectweb.asm.Type;
import org.objectweb.asm.tree.AbstractInsnNode;
import org.objectweb.asm.tree.ClassNode;
import org.objectweb.asm.tree.FieldInsnNode;
import org.objectweb.asm.tree.FieldNode;
import org.objectweb.asm.tree.InsnList;
import org.objectweb.asm.tree.InsnNode;
import org.objectweb.asm.tree.MethodInsnNode;
import org.objectweb.asm.tree.MethodNode;


public class DependencyFinder {

   public static void main(String[] args) throws IOException {
      if(args.length < 2) return;

      DependencyFinder df = new DependencyFinder();
      df.analyze(new File(args[0]), new File(args[1]), "org.apache.http/.*", "org.apache.commons.logging..*");
   }

   @SuppressWarnings("unchecked")
   public void analyze(File inputFile, File outputFile, String sClassRegex, String dpClassRegex) throws IOException {
      JarFile inJar = new JarFile(inputFile);
      JarOutputStream outJar = new JarOutputStream(new FileOutputStream(outputFile));

      for(Enumeration<JarEntry> entries = inJar.entries(); entries.hasMoreElements();) {
         JarEntry inEntry = entries.nextElement();
         InputStream inStream = inJar.getInputStream(inEntry);

         JarEntry outEntry = new JarEntry(inEntry.getName());
         outEntry.setTime(inEntry.getTime());
         outJar.putNextEntry(outEntry);
         OutputStream outStream = outJar;

         // Only process class files, copy all other resources
         if(inEntry.getName().endsWith(".class")) {
            // Initialize class reader and writer
            ClassReader classReader = new ClassReader(inStream);
            ClassWriter classWriter = new ClassWriter(0);
            String className = classReader.getClassName();

            // Check whether to process this class
            if(className.matches(sClassRegex)) {
               System.out.println("Processing " + className);
               // Parse entire class
               ClassNode classNode = new ClassNode(Opcodes.ASM4);
               classReader.accept(classNode, 0);

               // Check super class and interfaces
               String superClassName = classNode.superName;
               if(superClassName.matches(dpClassRegex)) {
                  throw new RuntimeException(className + " extends " + superClassName);
               }
               for(String iface : (List<String>) classNode.interfaces) {
                  if(iface.matches(dpClassRegex)) {
                     throw new RuntimeException(className + " implements " + superClassName);         
                  }
               }

               // Process methods
               for(MethodNode method : (List<MethodNode>) classNode.methods) {
                  Type methodDesc = Type.getMethodType(method.desc);
                  boolean changed = false;
                  // Change return type if necessary
                  Type retType = methodDesc.getReturnType();
                  if(retType.getClassName().matches(dpClassRegex)) {
                     retType = Type.getObjectType("java/lang/Object");
                     changed = true;
                  }
                  // Change argument types if necessary
                  Type[] argTypes = methodDesc.getArgumentTypes();
                  for(int i = 0; i < argTypes.length; i++) {
                     if(argTypes[i].getClassName().matches(dpClassRegex)) {
                        argTypes[i] = Type.getObjectType("java/lang/Object");
                        changed = true;
                     }
                  }
                  if(changed) {
                     // Update method descriptor
                     System.out.print("Changing " + method.name + methodDesc);
                     methodDesc = Type.getMethodType(retType, argTypes);
                     method.desc = methodDesc.getDescriptor();
                     System.out.println(" to " + methodDesc);
                  }
                  // Remove method invocations
                  InsnList insns = method.instructions;
                  for(int i = 0; i < insns.size(); i++) {
                     AbstractInsnNode insn = insns.get(i);
                     // Ignore all other nodes
                     if(insn instanceof MethodInsnNode) {
                        MethodInsnNode mnode = (MethodInsnNode) insn;
                        Type[] cArgTypes = Type.getArgumentTypes(mnode.desc);
                        Type cRetType = Type.getReturnType(mnode.desc);

                        if(mnode.owner.matches(dpClassRegex)) {
                           // The method belongs to one of the classes we want to get rid of
                           System.out.println("Removing method call " + mnode.owner + "." +
                                 mnode.name + " in " + method.name);
                           boolean isStatic = (mnode.getOpcode() == Opcodes.INVOKESTATIC);
                           if(!isStatic) {
                              // pop instance
                              insns.insertBefore(insn, new InsnNode(Opcodes.POP));
                           }
                           for(int j = 0; j < cArgTypes.length; j++) {
                              // pop argument on stack
                              insns.insertBefore(insn, new InsnNode(Opcodes.POP));
                           }
                           // Insert a constant value to repair the stack
                           if(cRetType.getSort() != Type.VOID) {
                              InsnNode valueInsn = getValueInstruction(cRetType);
                              insns.insertBefore(insn, valueInsn);
                           }
                           // Remove the actual method call
                           insns.remove(insn);
                           // Go back one instruction to not skip the next one
                           i--;
                        } else {
                           changed = false;
                           if(cRetType.getClassName().matches(dpClassRegex)) {
                              // Change return type
                              cRetType = Type.getObjectType("java/lang/Object");
                              changed = true;
                           }
                           for(int j = 0; j < cArgTypes.length; j++) {
                              if(cArgTypes[j].getClassName().matches(dpClassRegex)) {
                                 // Change argument type
                                 cArgTypes[j] = Type.getObjectType("java/lang/Object");
                                 changed = true;
                              }
                           }
                           if(changed) {
                              // Update method invocation
                              System.out.println("Patching method call " + mnode.owner + "." +
                                    mnode.name + " in " + method.name);
                              mnode.desc = Type.getMethodDescriptor(cRetType, cArgTypes);
                           }
                        }
                     } else if(insn instanceof FieldInsnNode) {
                        // Yeah I lied... we must not ignore all other instructions
                        FieldInsnNode fnode = (FieldInsnNode) insn;
                        Type fieldType = Type.getType(fnode.desc);
                        if(fnode.owner.matches(dpClassRegex)) {
                           System.out.println("Removing field access to " + fnode.owner + "." +
                                 fnode.name + " in " + method.name);
                           // Patch code
                           switch(fnode.getOpcode()) {
                           case Opcodes.PUTFIELD:
                           case Opcodes.GETFIELD:
                              // Pop instance
                              insns.insertBefore(insn, new InsnNode(Opcodes.POP));
                              if(fnode.getOpcode() == Opcodes.PUTFIELD) break;
                           case Opcodes.GETSTATIC:
                              // Repair stack
                              insns.insertBefore(insn, getValueInstruction(fieldType));
                              break;
                           default:
                              throw new RuntimeException("Invalid opcode");
                           }
                           // Remove instruction
                           insns.remove(fnode);
                           i--;
                        } else {
                           if(fieldType.getClassName().matches(dpClassRegex)) {
                              // Change field type
                              System.out.println("Patching field access to " + fnode.owner +
                                    "." + fnode.name + " in " + method.name);
                              fieldType = Type.getObjectType("java/lang/Object");
                           }
                           // Update field type
                           fnode.desc = fieldType.getDescriptor();
                        }
                     }
                  }
               }
               // Process fields
               for(FieldNode field : (List<FieldNode>) classNode.fields) {
                  Type fieldType = Type.getType(field.desc);
                  if(fieldType.getClassName().matches(dpClassRegex)) {
                     System.out.print("Changing " + fieldType.getClassName() + " " + field.name);
                     fieldType = Type.getObjectType("java/lang/Object");
                     field.desc = fieldType.getDescriptor();
                     System.out.println(" to " + fieldType.getClassName());
                  }
               }
               // Class processed
               classNode.accept(classWriter);
            } else {
               // Nothing changed
               classReader.accept(classWriter, 0);
            }
            // Write class to JAR entry
            byte[] bClass = classWriter.toByteArray();
            outStream.write(bClass);
         } else {
            // Copy file
            byte[] buffer = new byte[1024 * 64];
            int read;
            while((read = inStream.read(buffer)) != -1) {
               outStream.write(buffer, 0, read);
            }
         }

         outJar.closeEntry();
      }
      outJar.flush();
      outJar.close();
      inJar.close();
   }

   InsnNode getValueInstruction(Type type) {
      switch(type.getSort()) {
      case Type.INT:
      case Type.BOOLEAN:
         return new InsnNode(Opcodes.ICONST_0);
      case Type.LONG:
         return new InsnNode(Opcodes.LCONST_0);
      case Type.OBJECT:
      case Type.ARRAY:
         return new InsnNode(Opcodes.ACONST_NULL);
      default:
         // I am lazy, I did not implement all types
         throw new RuntimeException("Type not implemented: " + type);
      }
   }

}