Forget more megapixels, your next phone’s camera could offer whiter teeth

Qualcomm and MediaTek have both announced their new generation of flagship processors in recent months, which will power the high-end smartphones of 2023. In fact, we have already seen phones appear with these chipsets, such as the OnePlus 11, Xiaomi 13 series and the Vivo X90 series.

The processors bring more horsepower, hardware-based ray tracing and satellite connectivity, but it definitely seems more like an evolutionary year in terms of classic camera capabilities. Both the high-end chips from Mediatek and Qualcomm aren’t seeing major changes in terms of photo and video resolutions or frame rates.

But there’s more to camera support than just resolution, and indeed the two companies are rolling out quite a few imaging changes under the hood, such as professional video capture technology, optimizations for 200MP sensors, and native RGBW camera support. However, we also see a trend to unify AI and imaging hardware, and this will enable at least one rather interesting feature in 2023.

Qualcomm touts real-time semantic segmentation in the Snapdragon 8 Gen 2. For the uninitiated, semantic segmentation refers to the ability to identify specific objects and subjects within a frame. It’s a core technology that’s central to many camera modes because the camera software can identify specific scenes or people and then apply image processing.

Many smartphone brands use semantic segmentation for single-camera portrait modes, while others use it for AI scene recognition (sunsets, landscapes, flowers, food). We’ve even seen some brands like Xiaomi and Google tout the ability to completely change the sky, replacing a gray sky in your photo with a completely blue sky.

Read more: AI will help phone photos surpass DSLR, says Qualcomm

However, Qualcomm is drilling even deeper. First, the company confirms that its latest take on the solution is fast enough to run in real time and for videos. It also says its solution is able to identify specific elements such as teeth, hair, facial hair, lips, dust and more. And this could open the door to some interesting possibilities.

The most obvious is that we’ll see more accurate portrait mode snapshots. Qualcomm’s own video demo, in conjunction with ArcSoft, demonstrates the ability to more accurately blur challenging backgrounds (see below) while keeping trickier subjects sharp.

However, an intriguing possibility is that Snapdragon 8 Gen 2 phones could offer more detailed and advanced beautification effects. In fact, Judd Heape, vice president of product management for cameras at Qualcomm, told me android authority that the technology is initially aimed at selfie cameras.

We’ve already seen selfie cameras remove blemishes, smooth skin, and offer shape adjustments as beautification options, but this is just the tip of the iceberg with real-time semantic segmentation.

It’s also theoretically possible that we could see crazy beautification effects like teeth whitening. After all, this latest technology does indeed offer recognition of teeth. Heape agrees with this suggestion, explaining that partners can create their own semantic segmentation networks based on this technology to detect other things.

So yes, if you had a network that is really good at detecting teeth, that can be passed on to the ISP [image signal processor – ed], and the ISP can desaturate the colors in the teeth and change them from yellow to white. Absolutely, that’s definitely a possibility.

Qualcomm also touts its ability to recognize hair, saying this could be used to provide more detailed hair. But it also seems theoretically possible for brands to remove gray hair or the ability to completely change your hair color. Heape suggests that removing gray hair can be a tough challenge, especially if it’s just a few gray hairs in a sea of ​​dark hair. But he still thinks a complete hair color change is a possibility, though it may not look realistic.

However, manufacturers will have to walk a fine line between providing beautification features people want and promoting twisted beauty standards. After all, we’ve seen many questionable effects and filters over the years, such as face thinning, nose shaping, skin lightening, and eye dilation.

However, more advanced semantic image segmentation is not limited to beautification. The technology can also enable better processing of clothing, as Qualcomm’s video shows, providing extra sharpening for your jersey or jacket without affecting the rest of your body. The clip even shows the ability to remove glare from glasses.

Qualcomm also confirmed that real-time semantic segmentation is also programmable. So companies can use different neural networks if they have other uses for the technology in mind.

It’s all well and good for chipmakers to support something like real-time semantic segmentation, but the real question is whether smartphones will actually ship with this technology. After all, smartphone brands have a mixed record when it comes to using a chipmaker’s camera features (unlimited 960fps slow-motion, 120fps 12MP burst mode, for example).

Fortunately, Heape confirmed that this feature was available “right out of the box” for all smartphone brands. “So there’s no licensing fees, there’s nothing else the OEM has to do,” he explained.

So when we come out in 2023 (sic) there will be multiple handsets with this feature, quite a standout one.

In other words, this will not just be a theoretical feature, but one that will arrive on commercial devices in 2023. So keep an eye out for future launches from the likes of Samsung, Xiaomi, Oppo and other brands to see if real-time semantic segmentation makes an appearance there.

Qualcomm’s semantic segmentation improvements are possible thanks to the company’s Hexagon Direct Link feature. This refers to Qualcomm effectively creating a link between the AI ​​silicon and the ISP responsible for camera processing. Mediatek is taking a similar route with the Dimensity 9200 chipset, saying it’s fused AI and ISP hardware for more efficient 8K/30fps and 4K/60fps with electronic stabilization. Meanwhile, Google’s semi-custom Tensor chips in Pixel phones also use AI silicon that’s closely tied to the imaging pipeline.

This unified AI/ISP approach from Qualcomm and Mediatek in particular means that camera data can bypass relatively slow RAM, enabling more real-time camera processing. Fast processing doesn’t simply mean less time spent looking at a “processing screen” before previewing a photo, but it could potentially give us live viewfinder previews of different modes, new photo modes and new video features.

Qualcomm and/or Mediatek already promise advanced camera features in their latest SoCs, such as better bokeh video, video super resolution, photo blur, and better low-light performance. But it’s not hard to imagine future features like more detailed and powerful AR filters, Magic Eraser functionality for videos, burst mode with HDR for every shot or multi-frame processing for 50MP or 108MP shots with full resolution.

We even got our first taste of what’s possible when cameras bypass traditional RAM with the 2017 Sony Xperia XZ Premium. This phone featured a camera sensor with its own dedicated DRAM, enabling native 960fps super slow-motion video for the first time used to be. So we’re curious to see what else is possible with a much faster camera processing pipeline.

Thankfully, this unified approach to AI and ISP hardware won’t be exclusive to flagship devices, as Heape confirmed we can expect the feature to eventually land in mid-range chipsets at some point.

It’s interesting to see both Qualcomm and Mediatek come to the same conclusion about unifying AI and imaging hardware. And there’s no question that this could be the foundation for future smartphone camera developments. So while it doesn’t seem like there are many standout camera features in today’s high-end chipsets, these chips still bring significant improvements.

In saying this, we’re especially intrigued by this latest step in image segmentation. In addition to more accurate portrait modes, more detailed image processing, and improved beautification, real-time semantic segmentation already enables some interesting features. But we’re curious to see what else OEMs will come up with thanks to this mode and a more unified approach to AI and imaging hardware.