I'm using flutter and had to download xcode to build the app so I guess it's a necessary step.

I'm using flutter too, I've found a website called codemagic, so I can make builds for iOS. My laptop is windows. The bulk of development is building for Android.

I feel like most Xcode free-swift setups still secretly depends on xcodebuild for anything real.

It's nearly impossible to get away from Xcode entirely. You will always need the simulator and probably some entitlement / asset tools. What you can do is write the bulk of your app as an SPM package using your editor of choice, and then include that as a local or GitHub repo dependency in your Xcode shell app (which has no code, just assets, preview assets and plist files)

I had xcede[0] setup that worked well.

[0] https://codeberg.org/luxmentis/xcede

I use Xcode, but just to run the app with Command-R so I can see log messages or stack traces if I need to drop it into the agent for analysis. Everything else is done by Claude Code / Codex.

I even read the source using Zed not Xcode :)

tauri 2 + rust + webview give you a native feeling if you don't need any Apple frameworks

> .. webview ..

That would make it an web app tho. Not sure if that's what OP wants when making Swift apps.