Fix "android:* not found" errors

[Sleeeee]

This PR aims to fix the problem discussed in #3533. This effectively solves it for all the scenarios I've encountered/recreated, but please let me know if there are edge cases, I'd be happy to help.

To summarize, when resource namespaces are stripped, Apktool sometimes resolves the wrong namespace. The resolving passes the index of the attribute to getNonDefaultNamespaceUri() which fetches a namespace from the pool. When namespaces are stripped from the resource XML files, app attributes can be wrongly decoded as android: in specific conditions.

The suggested fix is, for app resources (nameId.pkgId() == 0x7f), instead of calling getNonDefaultNamespaceUri(), we can safely return ANDROID_RES_NS_AUTO to resolve the namespace.

To recreate the issue, we can use this hello world app. The source code contains this resource at res/drawable/trap.xml :

<?xml version="1.0" encoding="utf-8"?>
<selector
    xmlns:test="http://schemas.android.com/apk/res-auto"
    xmlns:android="http://schemas.android.com/apk/res/android">

    <item android:state_enabled="true" test:is_obfuscated="true" />
</selector>

Corresponding binary after building :

$ unzip strippy.apk -d base
$ mv strippy.apk base
$ xxd base/res/drawable/trap.xml
00000000: 0300 0800 f001 0000 0100 1c00 d800 0000  ................
00000010: 0800 0000 0000 0000 0001 0000 3c00 0000  ............<...
00000020: 0000 0000 0000 0000 1000 0000 2000 0000  ............ ...
00000030: 2a00 0000 5400 0000 8100 0000 8800 0000  *...T...........
00000040: 9300 0000 0d0d 7374 6174 655f 656e 6162  ......state_enab
00000050: 6c65 6400 0d0d 6973 5f6f 6266 7573 6361  led...is_obfusca
00000060: 7465 6400 0707 616e 6472 6f69 6400 2727  ted...android.''
00000070: 6874 7470 3a2f 2f73 6368 656d 6173 2e61  http://schemas.a
00000080: 6e64 726f 6964 2e63 6f6d 2f61 706b 2f72  ndroid.com/apk/r
00000090: 6573 2d61 7574 6f00 2a2a 6874 7470 3a2f  es-auto.**http:/
000000a0: 2f73 6368 656d 6173 2e61 6e64 726f 6964  /schemas.android
000000b0: 2e63 6f6d 2f61 706b 2f72 6573 2f61 6e64  .com/apk/res/and
000000c0: 726f 6964 0004 0469 7465 6d00 0808 7365  roid...item...se
000000d0: 6c65 6374 6f72 0004 0474 6573 7400 0000  lector...test...
000000e0: 8001 0800 1000 0000 9e00 0101 9b00 037f  ................
000000f0: 0001 1000 1800 0000 0400 0000 ffff ffff  ................
00000100: 0700 0000 0300 0000 0001 1000 1800 0000  ................
00000110: 0400 0000 ffff ffff 0200 0000 0400 0000  ................
00000120: 0201 1000 2400 0000 0400 0000 ffff ffff  ....$...........
00000130: ffff ffff 0600 0000 1400 1400 0000 0000  ................
00000140: 0000 0000 0201 1000 4c00 0000 0600 0000  ........L.......
00000150: ffff ffff ffff ffff 0500 0000 1400 1400  ................
00000160: 0200 0000 0000 0000 0400 0000 0000 0000  ................
00000170: ffff ffff 0800 0012 ffff ffff 0300 0000  ................
00000180: 0100 0000 ffff ffff 0800 0012 ffff ffff  ................
00000190: 0301 1000 1800 0000 0600 0000 ffff ffff  ................
000001a0: ffff ffff 0500 0000 0301 1000 1800 0000  ................
000001b0: 0400 0000 ffff ffff ffff ffff 0600 0000  ................
000001c0: 0101 1000 1800 0000 0400 0000 ffff ffff  ................
000001d0: 0200 0000 0400 0000 0101 1000 1800 0000  ................
000001e0: 0400 0000 ffff ffff 0700 0000 0300 0000  ................
000001f0: 0a                                       .

We can strip the namespace for is_obfuscated at offset 0x17c :

00000170: ffff ffff 0800 0012 ffff ffff ffff ffff  ................

After replacing the file in the APK and realigning/resigning, it installs perfectly and does not break.

$ zip -0 -u strippy.apk res/drawable/trap.xml  
updating: res/drawable/trap.xml (stored 0%)
$ java -jar $UBERAPKSIGNER -a strippy.apk
...
Successfully processed 1 APKs and 0 errors in 0.48 seconds.
$ adb install strippy-aligned-debugSigned.apk
...
Success
Install command complete in 434 ms

When decoding with apktool however, the attribute will be prefixed with android::

$ apktool d strippy-aligned-debugSigned.apk -o wrong
$ cat wrong/res/drawable/trap.xml
<?xml version="1.0" encoding="utf-8"?>
<selector
xmlns:test="http://schemas.android.com/apk/res-auto" xmlns:android="http://schemas.android.com/apk/res/android">
    <item android:state_enabled="true" android:is_obfuscated="true" />
</selector>

When rebuilding, we get the infamous error:

$ apktool b wrong -o strippy-crash.apk            
I: Using Apktool 3.0.2 on strippy-aligned-debugSigned.apk with 4 threads
I: Smaling smali folder into classes.dex...
I: Smaling smali_classes2 folder into classes2.dex...
I: Smaling smali_classes3 folder into classes3.dex...
I: Building resources with aapt2...
W: wrong/res/drawable/trap.xml:4: error: attribute android:is_obfuscated not found.
W: error: failed linking file resources.
Exception in thread "main" brut.androlib.exceptions.AndrolibException: brut.common.BrutException: Execution failed (exit code = 1): [/tmp/aapt2_105112275398347049556835392307132139706.tmp, link, -o, /tmp/APKTOOL10589230757756467911.tmp, --manifest, wrong/AndroidManifest.xml, --min-sdk-version, 21, --target-sdk-version, 34, --version-code, 1, --version-name, 1.0, --package-id, 127, --no-auto-version, --no-version-vectors, --no-version-transitions, --no-resource-deduping, --no-compile-sdk-metadata, --warn-manifest-validation, -I, /home/kali/.local/share/apktool/framework/1.apk, wrong/build/resources.zip]
        at brut.androlib.res.AaptInvoker.invoke(SourceFile:186)
        at brut.androlib.ApkBuilder.buildResources(SourceFile:357)
        at brut.apktool.Main.main(SourceFile:92)
Caused by: brut.common.BrutException: Execution failed (exit code = 1): [/tmp/aapt2_105112275398347049556835392307132139706.tmp, link, -o, /tmp/APKTOOL10589230757756467911.tmp, --manifest, wrong/AndroidManifest.xml, --min-sdk-version, 21, --target-sdk-version, 34, --version-code, 1, --version-name, 1.0, --package-id, 127, --no-auto-version, --no-version-vectors, --no-version-transitions, --no-resource-deduping, --no-compile-sdk-metadata, --warn-manifest-validation, -I, /home/kali/.local/share/apktool/framework/1.apk, wrong/build/resources.zip]
        at brut.util.OS.exec(SourceFile:147)
        at brut.androlib.res.AaptInvoker.invoke(SourceFile:183)
        ... 2 more

With this patch, resources will be correctly decoded and prevent apktool from crashing in those conditions :

$ apktool d strippy-aligned-debugSigned.apk -o right
I: Using Apktool v3.0.2-5-5f58cb98-SNAPSHOT on strippy-aligned-debugSigned.apk with 4 threads
I: Baksmaling classes.dex...
I: Baksmaling classes2.dex...
I: Loading resource table...
I: Baksmaling classes3.dex...
I: Decoding value resources...
I: Loading resource table from file: /home/kali/.local/share/apktool/framework/1.apk
I: Decoding file resources...
I: Generating values XMLs...
I: Decoding AndroidManifest.xml with resources...
I: Copying original files...
I: Copying unknown files...

$ apktool b right -o restrippy.apk                  
I: Using Apktool v3.0.2-5-5f58cb98-SNAPSHOT on strippy-aligned-debugSigned.apk with 4 threads
I: Smaling smali folder into classes.dex...
I: Smaling smali_classes2 folder into classes2.dex...
I: Smaling smali_classes3 folder into classes3.dex...
I: Building resources with aapt2...
I: Building apk file...
I: Importing unknown files...
I: Built apk into: restrippy.apk

fixes: #3533

[iBotPeaches]

thanks and appreciate the detailed steps! What I hit prior to this was when I was exploring multi-package apks - defaulting everything to a new namespace led to different issues. I was trying to identify a method of "picking/inventing" a namespace name that we can consistently resolve knowing the pkgId (or name). That way across many files we can devise and invent the same prefix.

@IgorEisberg may have a bit more modern knowledge at this than me to voice his thoughts.

[IgorEisberg]

I was trying to identify a method of "picking/inventing" a namespace name that we can consistently resolve knowing the pkgId (or name). That way across many files we can devise and invent the same prefix.

You can't, each binary XML is independent of others. Even if one doesn't define a prefix for a certain namespace, you can't rely on other XML files to fill the gap (i.e. "normalize" the namespaces).
In fact, apps like Settings.apk use both app and settings prefixes in strange ways.
Sometimes it's xmlns:app="http://schemas.android.com/apk/res-auto".
Sometimes it's xmlns:settings="http://schemas.android.com/apk/res-auto".
Sometimes it's even xmlns:settings="http://schemas.android.com/apk/res/com.android.settings".
In a few both are used: xmlns:app="http://schemas.android.com/apk/res-auto" xmlns:settings="http://schemas.android.com/apk/res-auto".
So, yeah, each binary XML defines its own namespaces independently.

@IgorEisberg may have a bit more modern knowledge at this than me to voice his thoughts.

I just kept the same behavior because I never stumbled upon all possible edge cases - didn't want to break anything, so I don't know much about stripped namespaces at the moment.

[iBotPeaches]

I was thinking of the normalization based on the pkg name, since we do know which pkg each XML is from. So stuff like com.example.pkg and com.example.notifications would be namespaces of pkg and notifications if stripped.

For this PR - I think we could expand test suite for this test and store the source sample app in https://github.com/iBotPeaches/TestApks

[Sleeeee]

Thanks for the feedback ! When the namespaces are wiped, the only real value you can hang on to is the pkgId, so maybe it would be possible to use that as a reference and create pkgId: namespaces ?

One issue is, because XML files are indeed independent, we can't be sure that all files either do or do not have namespaces. Maybe apktool could keep a global table of namespaces (and whether it would have to invent any) mapped to the corresponding pkgId ?

[IgorEisberg]

@iBotPeaches What you're suggesting is really messy when using a PullParser -> Serializer copy flow.
By the time you hit an attribute with a stripped namespace, where you only have a resource ID of the attribute and can infer the package name, the serializer has already written the xmlns:prefix="uri" attributes of the root node. In that case, injecting new prefixes would be only possible locally on that current node, which means repetitions.
Rough example:

<LinearLayout ...
  xmlns:android="http://schemas.android.com/apk/res/android">
    <com.example.CustomView ... somepkg:someAttr="..."
      xmlns:somepkg="http://schemas.android.com/apk/res/com.example.somepkg" />
    <com.example.CustomView ... somepkg:anotherAttr="..."
      xmlns:somepkg="http://schemas.android.com/apk/res/com.example.somepkg" />
    ...
</LinearLayout>

This quickly turns into a mess.
Even implementing that kind of "inventing prefixes on the fly" thing is a mess, because the parser's getNamespace* methods are hardlinked to NamespaceStack that keeps indexes into the string pool.
Even the current method of return ResXmlUtils.ANDROID_RES_NS_AUTO; makes a huge assumption that the "http://schemas.android.com/apk/res-auto" URI was previously given a prefix in that file, but in a real obfuscation case, it could have been stripped as well.

[IgorEisberg]

Thanks for the feedback ! When the namespaces are wiped, the only real value you can hang on to is the pkgId, so maybe it would be possible to use that as a reference and create pkgId: namespaces ?

One issue is, because XML files are indeed independent, we can't be sure that all files either do or do not have namespaces. Maybe apktool could keep a global table of namespaces (and whether it would have to invent any) mapped to the corresponding pkgId ?

I showed an example why relying on a "global pool of namespaces" is nonsensical and can never be trusted.
In my example, xmlns:settings had different URIs across different XML files.
It's a dirty hack, and a risky one.

[Sleeeee]

Added my test APK and a test case to the suite (let me know if you would like more tests).

As for the multi-package topic, if I understand correctly, this is not something Apktool does as of now (resolving non-sys non-auto namespaces if wiped). This PR does not aim to fix the multi-package resources, but hopefully at least makes a step in the right direction.

When trying to solve the bug caused when trying to resolve the 0x7f pkgId by assigning NS_AUTO unconditionally, it should not affect multi-package resources because no matter the package count and the names given to namespace declarations, pkgId 0x7f will always correspond to NS_AUTO (unless I'm missing something which is not unlikely).

Also, the current implementation uses the attribute index as input to determine the correct namespace, but I can't figure out why that logic would work, and why the attribute index should have an impact on the chosen namespace. As far as I can tell this is a bug, but maybe there is some structure/logic that I'm not aware of?

Let me know what you think, I'd be happy to help ! Thanks again for the input.

[iBotPeaches]

Merge if you are happy @IgorEisberg

[IgorEisberg]

Merge if you are happy @IgorEisberg

Frankly I don't know, you had that function for a reason and I haven't seen all possible edge cases, so it's up to you.
As for the test... that's a very strange way of working with XML. We have examples of doing that with xPath. But that can be tweaked later.

It's up to you, I'm not confident enough here.

[Sleeeee]

I have collected APKs from AXML-related issues to potentially find edge cases. I tested decoding-rebuilding APKs available from issues #1157, #2615 and #2981 and did not encounter any problem there. The APK from issue #2531 however, did throw a not well-formed (invalid token) error, which Apktool 3.0.2 also stumbles on, but as far as I'm aware, is unrelated to the namespace topic.

Also, I have been doing some research on that getNonDefaultNamespaceUri() function's history :

• 3ab1a1a from 2020 updates the function to start using the attribute's index as a parameter.
• 7d22fc6 introduces the function. From what I understand, back in 2017, the reason this tries to resolve the non default URI is because it wasn't standardized to the auto namespace as it is today ? I definitely haven't been around for anything close to as long as you guys, so my understanding might be off. Good to note that this is where the test apk from issue 1157 originates from.

It's of course up to you guys, but since we have a test case for this flow, maybe it makes most sense to merge and make sure that if edge-cases pop up we create proper tests for those too and review the logic? While the APK in the test case is created as a PoC, I can link to publicly available APKs that show this behaviour if needed.

[iBotPeaches]

Okay, I like the research provided and I wrote a lot of this so long ago when I was younger and dumber. I'll merge it tonight and update changelog.

Thanks for the research and fix @Sleeeee