Debugging my bizarre macOS memory issue in Swift

I was seeing some strange behaviour in the memory consumption of an app I’m working on. Being the Swift newbie I am, here’s how I figured out what’s up. Spoilers: There was no bug and computers are mystical entities.

Act 1: The problem

So I’m currently working on an app for macOS that has to bulk-load image files the user provides into CIImage instances to make some modifications to each image.

Thing is, it appears as though Core Image does something interesting when you tell it to load a bunch of images to memory and then release them again: Even after releasing all CIImage variables, memory usage of the app stays at a ludicrous level.

After processing & releasing all image files, app memory remains at 444mb

Having bashed my head against a wall with this problem for a bit, I decided to figure out the simplest code that could trigger this issue, which lead me to this code snippet here:

@IBAction func loadImages(sender: Any?) {
    let panel = NSOpenPanel()
    panel.allowsMultipleSelection = true
    panel.allowedFileTypes = [kUTTypeImage as String]
    let result = panel.runModal()

    if (result.rawValue == NSFileHandlingPanelOKButton) {
        for url in panel.urls {
            autoreleasepool {
                var image = CIImage(contentsOf: url)
                print(image?.extent.width)
            }
        }
    }
}

So far, it seems that just loading and releasing bunch of CIImages causes the issue. So my code dealing with image manipulation is off the hook for this memory mess for now. Off to Stackoverflow I went, full of hopes & dreams of an easy fix, provided by some wizards on the internetz.

Act 2: Instruments

Fast forward about a week: Not a single useful response to my Stackoverflow question. Clearly I had come across an issue so advanced, not even wizards could help me here. Dang it.

So, having read some articles posts and guides to understand how to debug apps with interesting behaviours in terms of memory usage, let’s have a look at what Instruments’ Allocation & Leak-detection tools have to say about this.

There appeare to be no leaks from our Applications... and the memory usage of the app is much lower than I'd expect. WTF!?

So, uh… this is interesting. According to Instruments, there are no Memory Leaks that originate from the app & it’s code itself. Even more interesting: Instruments reports a memory usage of ~30MB, while Activity Monitor reports well over 400MB of used RAM. WTF?

Okay, this seems strange. But, being a newbie to a lot of the concepts at play here, maybe I’m just misunderstanding stuff. Better do some more reading on the topic and get back to it then.

Act 3: What. The. Fuck

So I got distracted took a tactical pause before throwing myself at a few more articles on how Swift deals with memory management and I noticed something that’s even stranger than the numbers from Instruments’ Leak & Allocation tools:

When I let the app sit idly in the background, the amount of consumed memory begins to shrink.

To make sure this was actually what was happening here, I started a new build of the app, did a few “runs” of loading images & let the app sit idly in the background, while watching some excellently nerdy entertainment. And sure enough, memory usage starts at well over 400MB after the runs and miraculously dips to ~80MB.

Memory usage miraculously goes down when doing nothing

What. The. Fuck!?

Contemplating whether or not my MacBook’s RAM might just have been cursed by some kind of malign Chromium-devil, I handed a build of my snippet over to a friend. You know… just to sanity check against demons & stuff eating my RAM. My friend came back with the same results as I did: The app’s memory footprint decreases over time when actively using the machine.

Act 4: My knowledge of memory is bad & I should feel bad.

After far too much time spend googling for a specific Core Image issue, I now searched for a more general, “Why do Instruments and Activity Monitor disagree so much” issue. And sure enough: The wizards of Stackoverflow did have an answer. From 2011.

Activity Monitor is useless for development/debugging purposes.

[…]

Note that a system not under memory pressure will often not request that applications give back memory. That is, you may not see a drop in Activity Monitors numbers.

Ah.
So my app was, in fact, not consuming egregious amounts of memory. Activity Monitor was just displaying a different measurement from the system that makes it seem like there’s an outrageous amount of RAM being taken up when really macOS is just not asking for that RAM back.
That’s… good to know.

Act 5: Conclusion

So, what did we learn this week?

  • My app is actually fine. Memory usage isn’t nearly as much of an issue as I had feard
  • Don’t use Activity Monitor / Xcode as a measure of performance / RAM usage. There’s a tool for that. It’s called Instruments.
  • The RAM reported by Activity Monitor may not necessarily reflect the actual state of the apps running on my system. (Although, toggling on “Real Mem” to be part of Activity Monitor’s Table View sure can’t hurt.)
  • The people who work on this kinda low-level code are far too smart for me. Like, really way too smart. I’m talking “Getting a USB A connector attached on the first try”-levels of smart here.

I hope that maybe, down the line, this post might help someone with a similar problem to figure things out much quicker than I was able to. Or that it at least provided some entertainment to those of you who’ve been shaking their heads the entire way through this article because of how obvious my issue was.

If you’ll excuse me, I’ll be picking up some more Ibuprofen from the pharmacy. Banging my head against Xcode for a week really does cause some interesting kinds of headaches, let me tell you.

PS: While working on this “issue” I had found a blog post that explains macOS’ way of allocating, freeing & providing RAM in advance in much greater detail. Sadly, I have not been able to find this post again. If someone here knows what that post might have been, eMail me so I can include a link here.

»


How to Unfuck your Xcode Developer-ID Notarization Uploads

Update (June 4th 2019):

Xcode Upload displaying an error reading: There was an error sending data to the iTunes Store. Scheduling restart shortly...

It appears as though the issue described in this blog post has been resolved sometime after publishing. Uploading apps for notarization using Xcode 10.2.1 and above generates a funky looking There was an error sending data to the iTunes Store. Scheduling restart shortly... error, followed by the upload succeeding without issue.

I have no idea whether this was fixed by Apple, my ISP or somewhere in between. I’m leaving the original article up, just in case anyone ever stumbles over this issue again.

Per Aspera ad Failure

I was having an issue with Apple’s Developer-ID Notarization Service for Mac Apps, which had all of my binary uploads getting stuck at the “Uploading Package to Apple Services…” section of Xcode’s uploading process.

An exasperated Tweet and a Stackoverflow question with zero answers later I found myself debugging the issue myself for three super “fun” days. Here’s what I found out:

Network issues: Here be dragons

Having re-installed Xcode & Developer Tools multiple times, used a separate code base for testing, nuked my macOS installation & checked to see that uploads to both the Mac & the iOS App Store were also failing there really was just one possible cause for this issue left. The most horrifying & confusing of all the possible causes in IT: The network.

Sure enough, the upload goes through without any issues as soon as I connect to a network that’s getting its internet connection from another ISP.

So as things stand, no ammount of router jiggery & firewall voodoo seem to work. Any upload from within a network served by my cruddy cable ISP is destined to fail in interesting ways.

Apple’s partial solution

Knowing that all uploads to Apple Services, even iOS App Store Connect, are failing and that we’re dealing with a network issue, it turns out that, as per usual, Stackoverflow has a solution.

If your uploads to App Store Connect are failing, you can simply use Application Loader to upload an archived build of your app.

Application Loader has the handy ability to disable transfer protocols that are causing you issues. It’s as simple as checking off a checkbox in the apps “Advanced” preferences.

Open Application Loaders Preference and, in the "Advanced"-Tab, disable "Aspera" & "Isigniant"

Sure enough, having disable the problematic protocols my uploads to App Store Connect now work.

But there’s a catch:
Application Loader only supports uploads to App Store Connect, not the Developer-ID Notarization Service for macOS Apps. And Xcode’s Developer-ID uploader does not have an option for disabling problematic protocols.

The solution: Using Little Snitch

So if you need to Upload your mac App to Developer-ID Notarization & you get stuck at the “Uploading package to Apple Services…” bit in Xcode, here’s what you do:

Using a copy of Little Snitch (the most amazing macOS Firewall application out there) we first need to figure out if you’re running into an issue with the upload protocol. So start an Upload to the Notarization service through Xcode and keep a watchful eye on Little Snitch’s Network Monitor. You should see a process called ascp making connections on port 33001 to various .apple.com endpoints. This indicates Xcode trying to upload your binaries using the Aspera protocol, which appears to be causing issues when used with cruddy cable ISPs.

Little Snitch Network monitor showing the ascp process making connections on port 33001

Now, it turns out that Xcode does support alternative protocols for upload & when it detects that its protocol of choice can’t establish a connection, it will instead choose an alternative fallback protocol that should work even with a cruddy cable ISP (why it can’t detect the Aspera connection failing when it gets stuck is beyond me).

So to make it very clear to Xcode that the Aspera protocol is not an option we create a New Rule in Little Snitch that blocks all Outgoing Connections on Port 33001 for Any Application. Same goes for Incoming Connections.

Now, since ascp is an Apple utility connecting to an apple.com domain, you may need to disable the default macOS services rule group in Little Snitch for this to be effective.

Creating new Rules in Little Snitch to block incoming and outgoing connections for Any Application on Port 33001

If you’ve gone through this firewall setup, you should now be able to re-try your upload. Xcode should now use a fallback protocol and the upload should succeed.

Successful Upload in Xcode

While I’m still not sure about why Apple won’t let developers manually disable problematic upload protocols & why Application Loader can’t support Developer-ID Notarization, after banging my head against an Xcode-shaped wall I’m mostly happy to have figured out this workaround.

If you know more about the whys & hows of this issue, feel free too Tweet at me or send me a carrier-pigeon eMail.

»


Unresolved Mac App Notarization Issue

How to *not* spend your Friday: Bashing my head against the wall that is Xcode, trying to figure out why it just flat-out refuses to upload a binary for notarisation and gets stuck. What kind of piece of $%%##@! software even is this 🤦🏻‍♀️

TL;DR: If you’re with an cable ISP that has issues with supporting Apple’s preferred way of uploading apps to Apple Services, you’re SOL when it comes to notarizing your Mac Apps. There simply is no way to get them to Apple Developer-ID for you (yet, I hope).

For everyone with an attention-span longer than that TL;DR, here’s how I got to that conclusion:

Earlier this week I was having some issues with Apple Developer ID services. I was trying to upload an early build of a new macOS app I’m working on to Apple for notarization, a “feature” which will become practically mandatory for most Mac Apps soon-ish and is already de-facto mandatory for my app, because I had just signed up for a Developer-ID account.

Whenever I upload a Mac app to Developer-ID notarization or the Mac App Store through App Store Connect, the upload gets stuck at the “Uploading Package to Apple Services…” part. Every. Single. Time.

Having now re-installed Xcode, checked all my developer tools & nuked my entire macOS installation twice, I think I have finally figured out what is causing this issue. It’s the most daunting, devilish origin of all computer problems: The network.

Let me explain: When you upload your app to Apple (regardless of whether it’s App Store Connect or Developer-ID notarization), Xcode uses one of three available transfer protocols for the upload: Signiant, Aspera, or DAV. Which of these methods is used is up to Xcode, you get no say in the matter.

This is problematic when your network acts up when it encounters one of these methods & Xcode then confidently decides to use that exact method.

So what’s happening in my case is that Xcode tries to upload my binaries using the “Aspera” protocol, my network has some undefined issue with that and I’m left with a broken upload. Great.

At this point, there’s good news and then there’s bad news. The good news is that, for App Store uploads, Stackoverflow has a solution:
Export your binaries (iOS and macOS) as signed AppStore binaries (.ipa for iOS, .pkg for macOS). Then use Apple’s Application Loader from Xcode’s Developer Tools to upload these packages. Application loader gives you the option (under Preferences, Advanced) to disable all transport protocols that are not supported by your network. This way, your uploads to App Store Connect should succeed.

Go to the "Advaned" tab in "Preferences" and disable all upload methods that are giving you issues in Application Loader

Now the bad news: If you want to upload your app for notarization with Apple Developer-ID, you’re shit out of luck. Application Loader only supports uploads to App Store Connect and Xcode does not have options for you to disable problematic upload methods. So for the time being if your network / ISP are giving you problems in uploading to Apple Services, you’re stuck.

I have reported this problem through Apple Developer Technical Support and I’m still waiting to hear back from them.

In the meantime, if any of you know of a potential fix: Let me know via Twitter or eMail.

»