Apple is dabbling in AI image-editing with an open-source multimodal AI mannequin.

Earlier this week, researchers from Apple and the College of California, Santa Barbara launched MLLM-Guided Picture Modifying, or “MGIE;” a multimodal AI mannequin that may edit photographs like Photoshop, primarily based on easy textual content instructions.

On the AI growth entrance, Apple has been characteristically cautious about its plans. It was additionally one of many few firms that did not announce any large AI plans within the wake of final 12 months’s ChatGPT hype. Nevertheless, Apple reportedly has an in-house model of a ChatGPT-esque chatbot dubbed “Apple GPT” and Tim Prepare dinner stated Apple can be making some main AI bulletins later this 12 months.

Whether or not this announcement consists of an AI picture enhancing instrument stays to be seen, however primarily based on this mannequin, Apple is unquestionably doing a little analysis and growth.

Whereas there are already AI picture enhancing instruments on the market, “human directions are generally too temporary for present strategies to seize and observe,” stated the analysis paper. This typically results in lackluster or failed outcomes. MGIE is a distinct method that makes use of MLLMs, or multimodal giant language fashions, to know the textual content prompts or “expressive instruction,” in addition to picture coaching information. Successfully, studying from MLLMs helps MGIE perceive pure language instructions with out the necessity for heavy description.

In examples from the analysis, MGIE can take an enter picture of a pepperoni pizza and utilizing the immediate, “make this extra wholesome” infer that “this” is referring to the pepperoni pizza and “extra wholesome” could be interpreted as including greens. Thus, the output picture is a pepperoni pizza with some inexperienced greens scattered on high.

In one other instance evaluating MGIE to different fashions, the enter picture is a forested shoreline and a tranquil physique of water. With the immediate “add lightning and make the water mirror the lightning,” different fashions omit the lightning reflection, however MGIE efficiently captures it.

MGIE is accessible as an open-source mannequin on GitHub and as a demo model hosted on Hugging Face.