Amazing article. It just goes to show how many problems they are solving just to work around bugs and limitations within Apple itself. I’d wager a guess that if Apple made a new Xserve with the same performance as a Mac mini but twice the price people would still buy them, because figuring out data center operations is expensive.
Apple still runs iCloud with Puppet managed unhardened desktop Linux distros last I heard with most devops engineers having plaintext God access to all of it from their individual engineer ssh keys.
They are one of the last companies we should be trusting with hosting infrastructure.
Any server with a shell and opensshd is a pet, and has no business being near sensitive production workloads.
Add to that aging and wildly insecure ssh-based centralized ruby based infra management solution and I can only conclude everyone invovled has been asleep at the wheel on industry best practices for a over a decade.
Production servers should be immutable once deployed, remotely attestable at all times, with quorum signed secret management that trusts no single admin, with every workload in a dedicated vm with only a kernel, reaper, and target daemon, under an inaccessible offline base OS with TPM gated full disk encryption that fails closed if the system is ever modified after first boot.
And all of that is the bare -minimum-. Unfortunately negligence is industry standard since security engineers that actually know how to do system hardening are crazy rare in the bay.
People love to throw out these buzzwords describing their ideal “maximum security” setup suitable for “sensitive production workloads” without ever defining what the threat model is.
The most grating thing is requiring every workload to be in a dedicated VM. You are throwing so much performance away. What kind of threat model would justify this? Usually there’s no threat model analysis at all.
Banning opensshd is again mostly a fool’s errand. It’s much more productive to design a system that writes the authorized_keys file read by opensshd. For example you can design a system that requires peer approval before your own key gets written to authorized_keys.
Any service that has the potential to attract state actors, or attackers with access to frontier AI, MUST have a threat model that trusts no single employee or workstation.
From there the spec I described writes itself. No one can ever have the ability to mutate a system once launched.
A new server spec should be defined in a signed change, it is approved with a signed review, it is compiled deterministically from source and signed by multiple independently controlled systems which both must match, and then given multiple signatures. Then any engineer can deploy those immutable images, other engineers can remotely attest the expected PCRs from the independent signed builds. Finally secrets can be released to the machine by an HSM that validates m-of-n approval signatures.
This is table stakes zero trust deployment and again, the bare minimum.
Humans manually sshing to production machines and running commands is a 90s way of doing sysadmin that is exactly why Binance was just compromised and a thousand orgs before it. Survivors bias is not a reasonable security posture.
Also with even nested virtualization on modern hardware the perf hit is minimal, though distinct servers for each workload is fine too. At -least- sandbox each workload with something like gvisor at a minimum. Linux kernel namespaces are not enough as evidenced by endless exploits.
I practice what I preach here too. I spent the last decade building this way for high risk orgs responsible for hundreds of billions of dollars in value. I also founded https://stagex.tools and co-founded https://distrust.co and https://caution.co to make it easier for others to follow these basic tactics. Much more to come soon on this. All FOSS of course.
We need adoption on these tactics to be as wide spread as TLS if the internet is going to survive an army of AI armed skiddies.
Apple does not brag about using Linux, and while I will not call anyone out directly because it is a great resource, resumes of current and former Apple DevOps/Security engineers describing what they worked on are quite telling.
All sorts of intel you can get about all sorts of companies that way.
Yeah but that’s just hearsay right? To reiterate my original point, I’m not disagreeing with you, however I’d love to see some documentation and/or hard evidence of these alleged shoddy security practices at Apple
What company publicly documents their shoddy security practices?
That is part of the entire problem in the industry. No standards and no accountability. Those with the purse strings only invest in security that is marketable or protects IP. Anyone that invested in proper remotely attestable provably quorum managed systems is going to be bragging about it and happy to show off the proof to earn trust from their customers. I do.
When no one is bragging about doing these things, it is because they are not doing them, and they just have a bunch of humans running around with god powers to execute any code they want on production systems or in binaries that will end up on your laptop.
To get a clue about their internal practices, lets see what they do when everyone is watching in public.
No signed commits, no signed reviews. Merges done via browser. One Github API token away from being supply chain attacked. Also put one of those recent usernames into "forgot my password" and you can verify they do not mandate hardware 2FA. One compromised email account and you get to make a PR as a rando, then merge it with an Apple account and huzzah, you now have code in a critical repo used internally at Apple. Also releases are not signed or reproducible and are just cut by github actions. Another API key you could use to be even sneakier.
Almost any community maintained Linux distro does way better than this. Inexcusable for an org of their resources.
If this is how poorly Apple handles supply chain integrity in public, why would anyone believe they do any of this internally. I have also talked with multiple current and former Apple employees close to such things off the record but obviously will not out which ones.
Respectfully, I think you're missing my point here. When someone presents a claim, I hope you understand that the onus is on the claimant to provide evidence of their claim. Extraordinary claims require extraordinary evidence. I could just as easily assert that aliens from Jupiter visited me yesterday afternoon for tea but I can imagine you would like to see some evidence before wholeheartedly agreeing with me. Again, I don't think you're necessarily lying about the "plaintext god-mode access", but I am hoping you'll show us some evidence; "trust me bro" isn't good enough
I -did- give you clear public evidence of severe gaps in their supply chain security on at least one heavily relied on Apple software library. Same is true of all their public repos and code.
What better evidence could I possibly provide?
Also see the other commenter in this thread who identified themselves as a former Apple contractor admitting Apple uses Jenkins. -Jenkins-.
> plaintext God access to all of it from their individual engineer ssh keys
Scroll up, I asked for evidence of this specifically. This is a pretty damning allegation and worthy of further examination given the security implications for many, many users. Frankly, whatever they're doing in unrelated public repositories is concerning in a different context but otherwise thoroughly irrelevant to this claim above
Puppet does not have quorum signing or approval support. By -design- it means one god server that ultimately trusts every ssh key of every admin to make changes to a fleet of -malleable- RedHat servers via ssh like it is 2005.
But for just one attack chain to humor you:
1. Malicious party gets access to one Apple engineer ssh key (Or literally just their email account since they sign nothing and the key could be swapped out temporarily)
2. They make a PR injecting a subtle malicious dependency to any of the core Apple component repos almost certainly used on prod servers on the Apple github.
3. They use their stolen key to merge the PR.
4. Code lands in next release and eventually lands in Apple production servers because nothing is signed so how would anyone know?
Sure for bulk storage, but at least as of a few years ago Apple controls a lot of their own metal for the trusted backbone. I would not be shocked if they are crazy enough to manage GCP with puppet too.
Back when I did a consulting job for Apple Retail Software Engineering, the first part was getting a Jenkins CI/CD server up and running for the people who developed and supported the tools that were used by Apple to communicate with the employees in the stores, provide trading for them, etc….
Think of it as a predecessor to Facebook, plus whatever training platform you like. One of these guys wrote a complete software simulator for the iPod Touch-based device they were using in all the stores to handle point-of-sale transactions, so that he could include that in the training.
Content was created on macOS. Content was consumed on iPhones and iPads. The backend servers ran on Linux. It was my job to get all that compiling on the Jenkins server sitting under my desk, instead of on their laptops. I had to make minor changes to each of those code bases to get that to work.
Their build processes were quite advanced for the times. Thousands and thousands of regression tests. Thousands of UI conformance tests.
Then I had to get that Jenkins server up and running on the in-house VMware based cloud they had, where they sucked up their entire last two runs of rackmount xserve hardware and turned it around for their own internal use.
I was given a VM and remote access, and then that was that. My job was done. My contract was over, and I moved back to Austin. But that was a few years ago.
I feel like the ideal would be a 2U, 8 mac mini with dual 10Gb ethernet, shared dual power supplies, actual datacenter heatsink layout would be such a great CI platform.
The very purpose of these hosted Mac instances is to help people not buy a Mac of their own. Any higher margins would have to compensate Apple for this cannibalisation.
Not really. The main purpose of this is to run CI that never was going to be a personal machine anyway. This is just making it difficult for devs to have automated testing of their software on macs.
That’s like saying Apple didn’t listen to the market when it deliberately avoided going all in on a solo AI play.
It’s not all about what you think consumers think they want. There are other more pressing factors, like ability to deliver and supply chain management which have strategic impacts in the longer term.
I have no idea if this is true, but I would not be surprised if Apple has its own unreleased server variant based on the Mac Mini or Mac Studio platform that is deployed in data centers for specific clients.
Ok, I work for NoMac cloud hosting, and it might be interesting how we do it, in short:
A 21U rack at a datacenter (currently 2 in EU). It holds three MyElectronics 1.25U steel mounts at 2U pitch (3 Mac Minis each, rear ports, open floor for the bottom intake, 18 slots in all). 12 M5 Pro 15-core/24 GB/1 TB/10GbE NoMac Sendbox hosts at two 6-vCPU/8 GB VMs each, four M5 Pro 48 GB NoMac Simulator hosts at about 20 phones each.
Power comes from two Netio PowerPDU 8KS units (8 × C13, per-outlet switching and metering over an API) on the A and B feeds. Hosts are split evenly across the two feed and connected with C14-to-C7 cords, and an APC AP4421 transfer switch feeds only the network gear. The rack peaks around 1.3 kW and averages about 0.3 kW.
The network is a MikroTik RB5009 holdin the only public IP, doing outbound NAT and inbound WireGuard only, plus a MikroTik CRS312 giving 10GbE to every Mac on a 1 Gbit/s uplink. It has three VLANs: management (PDUs, KVMs), hosts with port isolation so they can only reach the gateway, R2 and the public internet.
Every host is commissioned with FileVault off and pmset autorestartatconnect 1, autorestart 1 and sleep 0, with auto-login on the phone hosts, hostd as a LaunchDaemon, zero-touch MDM enrollmen through Mosyle, and updates deferred 90 days.
Basically that's it. Fortunatelly, we did not have supply problems, but we are also smaller.
Anything posted on X already should be assumed low credibility by anyone, but someone willing to actually pay Elon a subscription for a blue checkmark I immediately assume is an idiot.
Apple still lives in the world where software developed in IDE on a personal computer and shipped as a boxed software on a disquette with a licence key. Xcode cloud is a joke. All their automation is a joke. It’s not surprising even their own software development stuck because of their ecosystem because they use same tools, same cryptic software signing with their plists, p8, chains, certificates, lack of ui testing, horrible cpp, objc legacy. You just drowning in this environment. How I wish we have proper web app apis at last
I used to drown in this stuff in the when I first did IOS dev back in 2017. I recently got back into it and was pleasantly surprised at how quickly I could setup a ci/cd pipeline. What used to take hours figuring out certs, debugging build issues, I can now even test my build against prod without even deploying.
I'm not building anything complicated, just a iPhone / watch app, but when I do get stuck with env its codex to the rescue. IMO relative to my experience many years ago its amazing.
Codex is definitely the reason I'm able to debug ops issues so quickly. For instance, I introduced a watch app into my iphone project. The build failed in xcode cloud but didn't give a reason. I gave codex some context and had it scan the project and it found the problem almost immediately.
In my experience this is only partially true. Apple has come a long way and their code signing experience is leaps and bounds better than Android at this point. Their UI testing is actually pretty solid, and Swift is a wonderful language to program in. The IDE part is partially true, but to be fair they are opening things up to coding agents (they're using Claude Code themselves) so I feel like this is only going to get better from here.
> Apple has come a long way and their code signing experience is leaps and bounds better than Android at this point
"Apple has come a long way and has made a very inconvenient and complex process only slightly less inconvenient, and people somehow are happy and pretend Android sucks."
We moved off GL/GH CI for iOS apps and switched to XCode Cloud only because of the pain that Apple is w.r.t. to signing. This is the other way, and luckily we have LLMs now: https://docs.fastlane.tools/actions/match/
Whatever we pay in runner-minutes/month is the price of our sanity so we don't have to deal with the nightmare that is their signing/release process, and their stupid CLI/XCode tools. Not to mention that even XCode Cloud seems to have been coded and shipped without any kind of second pair of eyes (QA nowhere to be found), and the XCode UI and Web UI behave in totally different ways for the same input. Of course without any kind of proper explanation for any errors, as in Apple's fashion.
> their code signing experience is leaps and bounds better than Android at this point.
In which universe ? You generate a signing key, its password, configure the alias, the alias password and then just modify your gradle build script to say signingConfig.set(file("file/path.jks"))
Hell, if you want to go even further, give up on signing, use the debug signing certificate and let google sign for you with their own keys.
I've stopped counting how many issues you get on trying to build iOS apps because "whoops, sorry, xcodebuild just farted and can't sign"
I assume you hit some limitations if you try to use Xcode Cloud for a larger project, but for small-scale use it’s pretty great. The setup is instant and seamless, builds are fast, and you get a bunch of minutes with your developer membership, which has been fixed at $99 for as long as I can remember.
Android tooling is cross-platform and easily executed in your choice of CI provider with cheap Linux ARM minutes. Simulator-based UI tests are headless (there are headless tests for iOS but not macOS). It's not even remotely a close comparison.
No, the apple ecosystem will always hold a monopoly over stupid upper class people to fleece. Apple will say "jump" and Developers will ask how high (if they know what's good for their wallet)
It's going to be interesting to see how this plays out with their "no unapproved code" stance. The old excuse was that normal users don't know how to write code and so don't need to be able to run their own code.
But now normal users can have some AI thing write single-use code for them, and for relatively simple code like "show me all my pictures between these two dates that are in landscape rather than portrait" they'll usually get it right.
And now who wants a phone where you can't do anything like that?
You absolutely can. You just have to pay Apple their $99/yr for the privilege. I have an app I vibecoded running on my phone to do regexps against a phone copy of /usr/share/dict/words because I find the regexp practice to be more fun when faced with a word game compared to getting stumped. There's some restrictions, like no running your own JIT on user specified JavaScript, aka no making a web browser, but for most apps I dare say you won't run into the limitations trying to scratch an itch.
They might still be living in that world, but they also live in a world where they can do exactly $0 of R&D of actual rack server SKUs and still have people buying their high-end MBP SKUs at halfway-decent used car prices.
Sorry for the clearly vibe-coded website. I use this to build a few iOS/macOS apps for personal use.
It started a few years ago (pre-AI) when I pieced together a toolchain for cross-compiling C/C++/Fortran from a Linux host to macOS (used by my former employer at https://p3m.dev/client/#/).
While the article is chock full of technical details, your comment is lacking. The idea seems interesting, but what's missing is details like cross-compilation performance characteristics compared to native, challenges faced, handling xcode upgrades, taking your project for a spin against popular projects on GitHub that use MacOS runners, etc. Saying "here's some vibe-slop so Linux is good enough" just doesn't cut it.
My intent was to inform others that you do not need Apple hardware to build software targeting Apple platforms. This is a very prevalent belief as if Apple has some “magic” that cannot run on other platforms.
I did not/have not polished this project because it works for my use cases.
Again, the original version of the project is used by my former employer who uses it in their package manager.
For years, "code review" in iOS meant "email your patch set to a particular email address, which would format your patch and send it to a mailing list". That list would be populated with people who could approve or disapprove it.
The server that ran that conversion was a MacBook sitting in a pretty random "server" room that included MacBooks, iMacs facing a wall, and other Mac devices used as headless servers.
Funny, I have been using an old 2014 MacbookPro has my homeserver since last year. To be fair, I've installed Linux on it, replaced the battery and set it to only charge to 60% battery. With full disk encryption requiring me to unlock it at every reboot, the built-in display and keyboard comes in handy, and the battery serves as a UPS just in case. Laptops make very good servers for home use, I think.
> Under certain conditions, an external USB NIC stops working and acts as if it is not powered and shows no link. Trying another cable, another NIC, another switch port, no luck. But a quick visit to Network Preferences, deleting and recreating the “network service” in MacOS Settings brings it back to life
I too have this really annoying issue where my Mac decides a USB-C or Micro-SD peripheral is no longer readable if I am working through multiple SD cards and backing them up, not sure why. Would love to work at Apple for 2 weeks with Claude to fix it and then quit and come back home, tbh would love to just work at all sorts of companies to fix really obnoxious "never to be fixed" bugs, Discord's abysmal roles stack would be on my list as well.
What interests me is the miniaturization and thinness of the mainboard. Imagine if the mainboards were produced en masse and mated with a perfectly-fitting (liquid-cooled) heatspreader. That could be a very space-efficient blade server for very high density compute.
Chock full of technical details, thank you to the authors! Got a few takeaways for using my own mbp remotely.
But if you read between the lines it is fascinating what isn't said - about Apple's customer focus.
These folks have no advocate at Apple; they worked all this out on their own. At this scale with hardware from Framework or Lenovo or Dell, there'd be a 24/7 hotline with a human to reach.
With its market power Apple can ignore these people - let them suffer, as it were.
They really don't care about developers it seems. I've literally been trying to give them money for an Apple Developer membership for over a month. Ten support emails later, I'm still stuck. I'm starting to think the only way is to buy an existing iOS development company. Plenty of similar stories in their developer forums.
Fun fact! You'll also need a DUNS number if you're publishing Microsoft software as well these days. Otherwise it's preemptively marked as malware! Yay!
Apple and Google as well (I have one). In my case, the issue is not getting the enrollment approved but literally just paying for it. It's some bug in their system and support is absolutely useless. They just keep replying the same unhelpful canned responses, probably without reading my emails at all.
I run a mac mini server at home and it was quite a culture shock getting that thing to do anything like linux. The password retirement at boot is a hard gate and your server will just not do anything until you unlock it. GREAT feature for desktop and laptops, not so much for a server.
At least they added the ability to unlock through SSH in macOS 26, so you can SSH to unlock, then use screen sharing to do the graphical login. Used to just be broken until you could physically log in.
With that kind of airflow and consistent temperature I would suspect the failure rates to be incredibly small on those batteries - typically what kills them is extreme temperatures and high C charge/discharge at high temperature deltas (as measured from 25'C) which is where the cells have lowest degredation, which they should see very little of in this environment.
What would be more pertinent is changing the charging logic so the packs are held at 50% state of charge vs being always charged up to 100%, that will significantly decrease the chance of pack issues while greatly increasing the longevity of the battery itself. If you hold a lithium ion cell at 50% SOC for years without varying it and keep the temperatures flat, it can last for a very long time without capacity degredation. If you leave it at 100% it will be degrade in a few short years and can swell, at which point it has twice the chemical potential to flame out. Also if it's charged to 100% when the datacenter is at 20'C ambient and datacenter cooling fails and temps go up, having the pack warm up to 40'C can overvolt it and that is usually where they swell/fail.
Also fair to note here that Apple is probably at the frontier in terms of using the highest safety cells and battery management logic so I'd feel better about this than I would putting a lesser brand's lipo pouches in datacenter environment.
> With that kind of airflow and consistent temperature I would suspect ....
A lot of words, many of which might well be true. But it ignores the fundamental point.
I think we can safely assume its not their building, not their datacentre.
As a result, they need to play by the rules of the datacentre operator.
The datacentre operator has fire suppression systems build to a certain spec.
The datacentre has risk assessments and insurance covers to think about.
The datacentre has other customers to think about.
The datacentre has its own staff to think about.
Yes, datacentres have some of the most advanced multi-layer fire detection and suppression systems in the world.
But a fire caused by multiple racks stuffed full of LiPo batteries will likely result in very hard to control results, especially if it becomes a cascading event.
Yes, it is arguably highly unlikely to ever happen, especially with Apple hardware as you say.
However at the very least the guy should fess up and have a chat to the datacentre operator to at least give them a chance to say (preferably in writing) "ok, no problem, we'll just add it to our risk register".
We run lithium ion UPSs in our racks and have for many years - I think you're taking the airline approach here but the reality is the environment inside a datacenter is much more stable than almost everywhere else lithium ion batteries are employed today, and the fire suppression detection and suppression systems add on top of that.
When the MacBook is fully charged and still plugged in, it bypasses the battery and runs directly off power from the wall adapter. So those batteries are hardly doing anything.
> A well-known limitation is that Apple only allows two macOS VMs per physical Mac. It can be a Mac Pro with 24 cores and 192 GB memory, yet it is limited to running only two guest VMs. This limitation leaks into our product offering: we can only slice Mac resources in two halves. So we run instances with 6 or 12 cores on 14-core Mac Minis and 8/16-core instances on 18-core MacBooks (we reserve a couple cores for management overhead).
I don't know about selling, but Apple permits up to 2 virtualized Macs to run on a Mac. If they are circumventing this limit that would be an issue, but it doesn't sound like they are.
Yes but if you read the EULA the two VM limit it does sound like you can't really sell access to VMs:
> except as otherwise provided in writing, signed, or issued by an authorized representative of
Apple, to install, use and run up to two (2) additional copies or instances of the Apple Software,
or any prior macOS or OS X operating system software or subsequent release of the Apple
Software, within virtual operating system environments on each Apple-branded computer you
own or control that is already running the Apple Software, for purposes of: (a) software
development; (b) testing during software development; (c) using macOS Server; or (d) personal,
non-commercial use.
In section 3 they also say that if you're renting access to a Mac you can only run one VM, which sounds kinda like that
> Except as expressly permitted in Section 3, the grant set forth in Section 2B(iii) above does not permit
you to use the virtualized copies or instances of the Apple Software in connection with service bureau,
time-sharing, terminal sharing, relay service or other similar types of services.
Ah okay - and the 24 hour minimum rental time doesn't apply to VMs? Im just curious how they are getting around the limits that Mac Stadium and AWS have.
They aren't getting around any limitations. How you decide to bill for meeting the limitations is a business decision, and you can do it via either higher average rates and allow more granular rentals to be financially covered that way or by requiring a 24 hour minimum.
I.e. just because Apple requires you to do things in 24 hour increments does not mean your billing has to expose it in exactly the same way.
If a business reserves a Mac for 24 hours (and allows no other users for that window), it complies with the EULA, no?
That doesn't mean the business has to charge the user for 24 hours. You could charge for 6 hours and then just let the Mac sit idle and unused the remaining 18.
You're eatting the cost of that unused time, but if its a differentiator that leads to more business than it might still be worth it.
EDIT: To be clear, imagine a case where the user has 24 hour sole access to the Mac. They could use up 24 hours if they wanted. But usage is tracked per hour, and they're only billed after the fact based on how much time they actually were running something during that 24 hour window.
From the wording of the parent comment my understanding is that 24 hour period applies to the MacOS VM, not the hardware itself. Since they split each physical host into at least 2 different VMs the 24 hour period is for that VM. Even stronger case if they also lock the core affinity to the physical layout like Max editions having multiple Pro chips/dies within...
For Amazon EC2 the case is different since it doesn't allow VMs but only possible to get dedicated bare-metal hosts. Obviously Amazon could offer VMs but they would not because the current way allows them to charge 24 hours even if user uses the host only 1 hour. And obviously no body big enough challenging the status quo and impacting AWS' revenue hard enough.
The hack for the 24 hour thing is, of course, as a data center, to form an s-corp and hire employees to work 1 hour per week racking servers and give them a W-2 for a princely sum of just $16.90/week which works out to $878.80/year. Then as a perk of the job, those employees get to access the cluster as a perk of their employment. You can't charge an application fee since it's California, so then you sell interview prep courses for $1,000, and encourage graduates to apply.
GitHub Actions gets around the 24 hour rule by saying you agree to have them as a subcontractor, so you're paying them for their infrastructure and services, not for the minis themselves.
At TestingBot we’ve recently deployed Mac Mini M6 instances for (automated) testing on macOS using Virtualization.Framework.
Never considered Macbooks; what about the battery hazard in a rack? We also host iOS devices, after a couple of months of intense use, the batteries tend to swell up and bump up the screen of the device.
Yeah Apple needs to fix this by either making Xcode Cloud better, make more Mac minis (or just dedicated server hardware), or even loosening the two-VM limit. This kind of thing is bad for the average consumer and just plain wasteful.
They literally don't care. To them, either it's 1999 and your Macs should all be on desks at your physical office that you definitely have, or, you can "just" use Xcode Cloud[1], which is definitely fully featured and surely addresses the only possible 'headless' use case.
Plus, since the desperate are willing to do crazy-looking things like this, bulk-buying $6,000 premium laptops with batteries and screens, at enormous margins, Apple is even more incentivized to keep (not) doing exactly what it's (not) doing.
Amazon also has much tighter integration and offers the Macs bare-metal. This has the downside that you can only gets Macs as dedicated hosts, so it's another layer to manage, and that it takes around an hour to wipe the Mac whenever you shut it down, even if you're not going to release the underlying dedicated host. Also for the last year, even grabbing a Mac dedicated hosts often runs into "None available" for hours at a time. Amazon tells you "try this other zone" but even that is no guarantee. I wasn't even able to get our TAMs to tell me whether trying other regions was likely to help. Besides that, it takes 5-10 minutes to launch a Mac, and once you acquire one, there's a 24 hour minimum billing period due to Apple licensing restrictions. Which is all to say, as a single customer, you can't really build an elastically scaled Mac CI/CD farm on top of AWS. You really need to size for your peak load. Which is expensive compared to just buying your own Macs, but $former_employer didn't want to be in the datacenter business.
FWIW, I believe MacStadium was the first to offer Macs in datacenters as a service:
This is quite an amazing setup and shows the current limitations of Apple supporting their enterprise customers, which end up buying MacBooks to make their products available...
There've been many words spilled over Apple's... studied disinterest, in enterprise customers.
It's kinda hard to argue with their results, though. They handed the enterprise market to Microsoft 40 years ago, and have shown few-if-any signs of changing their mind.
the issue is the internal power supply is not redundant, even if you supply it with an ATS or UPS, that component can still die taking out the entire thing.
> the issue is the internal power supply is not redundant, even if you supply it with an ATS or UPS, that component can still die taking out the entire thing.
I mean, its basically the same thing in the MacBook, no ? The SMC ?
After all, there needs to be a magical chip inside to make sure you don't, for example, fry the laptop because it accepts charge via both USB-C and the MagSafe port.
Sure SMC failure is incredibly rare, but it doesn't mean the theoretical SPoF is not there.
Also when was the last time you had an internal PSU on a Mac fail ? Its been a very long time since I've ever seen one, and that's even thinking back to the old days with the old-school canister capacitors. Say what you like about Apple, but on the whole they do a decent job on their BOM selection.
"Luckily by now we found almost perfect cables that bend exactly as we want them to bend. And we bought the entire supply of them. There are zero more of them available to order on Amazon." Everyone is hoarding at any cost...
I'm a little surprised that people would rather rent a Mac than buy one. Less so now that Apple raised the prices but for a long time a M4 Mac Mini was $599 which only works out to about 6 months at these prices.
Especially since they're not like a regular server, loud and power hungry. They're the definition of quiet and power sipping. I have one do all my background coding work when I want to keep my laptop clear (being headless helps with me moving around too).
Dang, I hope they sell the older laptops for cheap (less than what they were at time of purchase obviously) on Ebay once they're phased out, wouldn't mind buying a "datacenter mac" if its still viable...
I could see them flooding the used Macbook market and selling like hot cakes, but also giving them back "rack" space.
We have all the hosts enrolled into MDM, so we have all sorts of access ready: SSH, KVM, VNC, MDM. With Macs in the DC we found having all these ways to access the machines is really essential. Things break in unexpected ways.
In terms of settings, besides the obvous (do not sleep, do not auto-update, etc), disabling USB restricted mode helps a lot (Privacy&Security > Allow accessories to connect) - no need to approve connected devices via UI.
... and I'm slooowly very slowly thinking of creating a "mock Docker".
Think 80% of Dockerfile syntax supported for running "docker build" against macOS guests. I do have some pieces already done here and there and basically it all comes down to actually creating a product out of a pile of various working scripts (OSS obviously). That's the hardest part.
And then comes the final boss of the game: marketing. Not sure anyone needs this at all.
This directly challenges the author's claim "You can't build iOS apps without a Mac."
omarchy-apple-dev is built on: https://xtool.sh which is "a cross-platform (Linux/WSL/macOS) tool that replicates Xcode functionality with open standards."
That stack is based on xtool at its core (xtool.io). It's very cool, but still early for steady enterprise use IMO.
Also it doesn't have iOS Simulator, so automated testing is out of question. I doubt that porting the Simulator would be feasible, since it requires macOS kernel.
Well, maybe with XNU in a VM and reimplementing parts of macOS userland. Would love to see someone try.
Also: which solution you have chosen for the p2p deployment? I used to manage deployments on a large scale cluster and got it down from 2h to <1m using libtorrent.
Yeah, as alluded to in the post, we have a custom p2p distribution, very inspired by BitTorrent. So you're on the right track :). We'll probably share more on that topic at some point too.
But honestly before we had that, we started with a system where nodes pulled whole images from each other (a decentralized algorithm without central coordination, but no chunking). It worked surprisingly well considering how simple the implementation was.
I was hopeful that they might lift the limit with a refresh of Mac Pro (it even had a rack mount version). But with Mac Pro discontinued not keeping the hopes up anymore.
I recently setup my home lab in a way that kind of replicates their end goals and it wasn't hard:
1. Beefy M6 Mini as the server. UTM managed MacOS VMs with screen sharing enabled. 1 VM per project. Set the Mini and VMs to never go to sleep.
2. Secondary Macbook Air -- I screenshare into the VM for the project I want to work on and do work from there. I can close the laptop at any time and not worry about agents stopping.
3. With Tailscale on all machines I can work from anywhere.
On a related note, has anyone modified a Mac Studio to run directly from DC power? I'd like to use one in a battery-powered system without an inverter.
It is mind blowing to me that a 1 trillion dollar company don’t have a way to support large customers with enterprise servers. Fun fact: they used to have.
Try now stacking ten iphones, or as many as you want. The world definitely will get more interesting in the very near future
Yesterday I saw AI in a pendrive, which got me wondering if small set-top boxes isolated from the web, connected to TVs or your local WiFi will be the norm of tomorrow
The cost of cloud Mac hardware is quite frankly ludicrous across the board. Namespace's minimum for an M6 is, what, $0.06/minute for 6 vCPU? And that's on the cheap end - Semaphore wants $0.09/minute for 4 vCPU and GitHub Actions wants $0.062/minute for M1 hardware (forget AWS pricing even which is even more atrocious).
If I run just a UI test suite for an hour, that's $3.60/run, which is now higher than what I'm paying for AI inference that made the change that triggered the run. That's crazy! If I ran even just one run per night that would run me more than $100/month. A Mac Mini is $900 - I break even in nine months by buying the hardware outright. If I run multiple UI test suites per day, I break even much faster. These ARM chips are so power-light that the cost of electricity is practically irrelevant, and after all, these aren't rack-mounted monstrosities with 90 dB fans - they're meant to cool themselves sitting on your desk. Maybe it's different if you need like 60 of them?
Is anyone even running the numbers before they sign up? Or are companies really running that many Minis and are paying the premium to outsource the DCOps?
I have a test suite that needs to run on a Mac in a project that merges < 20 PRs a year. No point in provisioning our own Mac and maintaining it for that.
Yep. I run a local CI cluster of Macs made up of old laptops, so the effective cost is "zero" (resale would be annoying and I probably wouldn't bother). It's pretty easy. No way am I paying those kinds of rates for cloud Macs.
I looked at our GitHub Actions monthly expenses, then at the price of a M4 Mac mini (just when they were released, and still cheap), and decided to buy one as it would pay for itself in less than a year. There are definitely still cases when buying hardware still makes sense, over renting in the cloud.
Mac Minis are small and power efficient. Scaling questions are always a question of degree. Several Mac Minis, Ethernet cabling, a KVM switch, maybe a simple MDM to make your life a little easier, scales just fine.
xtool [0] came up on here a while ago [1]. I think it still requires you get a big xcode app bundle off a macos machine and drop that into xtool, since there's no other way to get the iOS library header files from Apple. The other parts of xcode just aren't used as far as I know. Other than that caveat, you're running builds in xtool's own environment on whatever platform you choose. You can even do code signing.
That's one of the reasons to keep it as far as possible from anywhere you need trust (like prod): software, like soylent green, is made of people. And you can't trust people like him.
As an individual developer I would totally violate that part of the EULA, but if I'm selling OSX computer like OP, you're exposing yourself to much greater legal risk.
I could see Mac minis being used for data center type setups. But MacBooks are distinctly consumer devices. I know your company is trying to solve a niche. But if it stops becoming a niche, consumers will be impacted because of operations like yours.
The article addresses that. In the very beginning.
> But still, why laptops? Isn’t it a major waste of screen pixels and keyboard buttons and battery ions? Well yes, Mac Mini would be a better choice for a datacenter. Sure, it has very rudimentary remote management, no redundant power, and a beautiful case that ensures that cooling will be difficult regardless of which side you lay it on. But these are all known challenges that we have learned to manage.
> But have you actually tried to order a couple hundred Minis? We have. Ordered in January, delivered in July. Welcome to the world of 2026 when the supply chains are in disarray. Never mind that Apple doesn’t release updated CPU models to all devices at once. So Mac Mini didn’t get M5 cores for 10 months (and then suddenly got M6 before any other device model!). We promise the best and the newest to our customers so we had to be inventive. That’s how the MacBook-in-Datacenter project was born.
I really wonder if Apple's logic to require a sale of a unit for 2 VMs is fiscally correct. Seems like it would be better to open it up to virtualization such that more devs can make iOS and macOS apps. Is the community of devs being forced to buy mac hw just do dev a iOS/macOs app so large that its worth putting us through all this HASSLE? Maybe they are worried about clone machine/farms that will eat into their sales. Idk. Because they are so VM unfriendly, I refuse to deal with Apple's bs.
Because the target market (mobile) is very profitable, if you have the skill, I also don’t understand that stance because Mac powerbooks, Mac minis, Mac studios last a long time and have high resale value if you decide to move on, particularly if you are a smaller development company.
I am lost, doing it in a Hackintosh is feasible but against EULA terms? I remember having a Hackintosh around the office two years ago (yes, it was x64 based). BTW I just found "ARM is not a universal architecture, there will be no more Hackintosh" [1].
The registry distribution subject "for another blog post" is the most interesting part of this to me. I'm very interested in the efficiency/locality metrics that went into designing that. I look forward to that future post.
Apple bros, why don’t you just make servers? People would go nuts for a 1TB RAM Mac server. You’re a hardware company! Why are you throwing away one of the biggest hardware markets!?!?!
> But have you actually tried to order a couple hundred Minis? We have. Ordered in January, delivered in July. Welcome to the world of 2026 when the supply chains are in disarray.
I hope you guys are the only ones employing this strategy otherwise you're just contributing to the same problem mac minis have.
Many people/companies are willing renters today even when the resale value is very good with Mac’s in comparison to PCs. Never have trouble selling on decently equipped Mac computers. The PCs, aside from the graphics card, end up in a scrap heap. (computer graveyard).
