Apple Intelligence: The AI features coming to your iPhone

Apple is adding generative AI tools to its phones and computers that can rewrite and summarise text, create “cartoony” images, edit photos and more, as well as an optional integration with OpenAI’s ChatGPT.

The collection of tools, which the company calls Apple Intelligence, will arrive later this year for iOS 18 and MacOS Sequoia, which were unveiled at the annual Worldwide Developer Conference on Tuesday morning (AEST). They also include an upgraded and more natural Siri assistant, which now understands text prompts and can respond with step-by-step instructions.

Craig Federighi, senior vice president of software engineering at Apple, shows off Apple AI.

Many of the iOS tools, which will work on iPhone 15 Pro or later, are similar to those already available on Google and Samsung phones or through Microsoft apps such as Office. But Apple chief executive Tim Cook said the company was doing things differently.

“Apple Intelligence will transform what users can do with our products, and what our products can do for our users,” he said, in announcing the tools.

“Our unique approach combines generative AI with a user’s personal context to deliver truly helpful intelligence. And it can access that information in a completely private and secure way to help users do the things that matter most to them.”

Apple AI features will be available across Mac, iPad and iPhone.

The new tools include the ability to highlight any text and have it be rewritten in a different style, or summarise text from web pages or any app. Users will also be able to record audio and have it transcribed and summarised automatically, from either the Notes app or while on a phone call. For the latter, the other participants in the call will be alerted to the fact they’re being recorded.

Image generation will work in several applications but will also have its own dedicated app called Image Playground. Users can add prompts and descriptions to create images instantly, and can even choose people to include from their Photos library, but the output is always in a “cartoony” style rather than a realistic one.

In Messages the system will suggest custom stickers based on conversations, while in Notes a scribbled picture can be transformed into a more defined image. Users can also create custom emojis using text prompts.

In Photos, a new clean-up tool can remove distracting background elements with a tap, while users can also surface specific images by describing them.

Bringing all of the Apple Intelligence tools together is a redesigned Siri, which can be interacted with using voice or text. Apple said Siri would be able to look at a user’s screen for context or dig through their data, to accomplish complex tasks such as “Play that podcast that Jamie recommended” or “Send the photos from the barbecue on Saturday to Malia.”

Siri and Apple Intelligence can use Chat-GPT for some requests, such as generating or refining text.

Apple stressed that it was taking a privacy-first approach to AI, with as much data processed on-device as possible. Any cloud processing is done in Apple’s custom secure servers, which limit personal data collected. However, the system can also use external AI services, the first of which is OpenAI’s ChatGPT.

When a user asks for something that’s best handled by ChatGPT, the device will ask permission to share the request, and OpenAI will not log the user’s IP address or store their data unless they’re a paying OpenAI customer who has linked their account. The ChatGPT integration is free to use on Apple devices.

Vision Pro and other updates

Also at the conference, Apple announced that its Vision Pro headset would arrive in Australia on July 12, and would start at $6000. The so-called spatial computer is also getting its first annual software update with VisionOS 2, including the ability to turn regular photos into 3D spatial photos, and a new extremely wide-screen virtual display for use with Mac computers.

Apple’s Vision Pro headset is coming to Australia in July.

Aside from Apple Intelligence, iOS 18 was shown with a redesigned Home Screen that allows more customisable app layouts and the ability to tint app icons with any colour. An AI system will now also identify notifications, messages and emails it thinks are high priority to emphasise them above others.

The iPhone will also get the much-requested ability to send and receive RCS messages, which should mean better multimedia support between Apple and Android devices. A new Passwords app offers an Apple-focused option for a password manager, while users of recent AirPods models will be able to shake or nod their head to interact with Siri.

iPadOS is getting many of the same features, but a special mention was given to a new calculator app. Not only can it function like a regular calculator, but it will also work out and respond to sums you write by hand, any time you create an equals sign.

Apple TV is taking a cue from Amazon with a new feature called InSight, which shows you which actors are in a scene when you pause a show or movie. It will also display what song is playing, and let you add it to your Apple Music library.

Finally, a new continuity feature in MacOS lets you control your iPhone from your computer. While the phone stays locked, users are able to navigate the device and even use apps by interacting with a virtual version of their phone on their computer screen.

Get news and reviews on technology, gadgets and gaming in our Technology newsletter every Friday. Sign up here.