Synching Lucy's lips with TV sound By Junko Yoshida , EE Times April 19, 2004 (2:08 PM EDT) URL: http://www.eet.com/article/showArticle.jhtml?articleId=3D18901792 PARIS - In movie theaters of yesteryear, it was not uncommon for the audio and picture to be out of synch, with the silver-screened actors' mouths moving and sound coming a half-second or more later. It was amusing until it became annoying, and patrons invariably yelled out to the projectionist, who restarted the film. Technology eventually solved that problem, but in a bizarre piece of irony, the technology has gotten so sophisticated that the problem has resurfaced. TV makers have only recently begun to recognize the problem of audio information being asynchronous with lip movement on flat-panel TVs, realizing that it could become a bigger issue as sales of flat-panel displays start to take off on the consumer market. Average CRTs do not have audio and video synchronization problems because the time needed to process video and audio - thousandths of a second - is about the same, said Stefan Hepp, director of marketing of consumer audio at Micronas GmbH (Freiburg, Germany). But when advanced TVs, LCDs and plasma displays add enhanced picture-improvement functions like deinterlacing, motion-adaptive conversion and scaling, the result is video processing that takes more time than audio processing, he said. So audio processing time must be adjusted to match the video. Observers estimate that two-thirds of flat panels - except for low-end versions - will need a lip-synching device. Some display vendors are said to be unaware of the potential problem and some simply prefer to deny it. Others, forced to look more closely, have just started, reluctantly, to acknowledge the lip-voice lag. That could blow into a bigger issue among consumers, Hepp cautioned, after "consumer test magazines begin to make measurements on the audio and video asynchronous problem." Paul Martin, marketing manager for digital TV systems at Philips Semiconductors, agreed. Video-processing delays "started to become a problem on LCD displays recently," he said. While LCD displays could introduce video delays with 15- to 20-millisecond response times, memory-based video-processing algorithms, often used in mid-to high-end displays, could also add visible video delays, he explained. Combine the two and, "You could have a considerable delay in video - more than 100 milliseconds in some cases." While the large high-end PDP or LCD panel markets can certainly afford an additional chip or high-end SoC with audio delay functions, the real emerging battleground is the midrange LCD or PDP market, where the cost adder for a 30- or 40-inch flat panel is not trivial and where the technological problem (or cost of the solution) could stall a promising market. Micronas' Hepp cited "motion-adaptive conversion," "scaling" and "deinterlacing" as three main sources of video delays built into TV electronics. He also said that some plasma displays, which need to put data into internal memory before lighting the panel, could introduce 20 to 60 milliseconds of delay. Overall system-level video delay can vary among flat-panel displays, because system vendors often mix different types of scalers or deinterlacers. The industry does not agree, however, on the threshold at which audio/video asynchronization becomes a problem. How many milliseconds will it take a consumer to notice that Lucy's lips aren't catching up to her words? And how long before the consumer gets irked? A test by a German university late last year found viewers began to notice things were "a little strange" at 20 ms, said Hepp. A 30-ms delay began to irritate them. Beyond 30 ms, they said, "I don't want to have this on my TV." Meanwhile, Philips' Martin believes the delay only becomes noticeable at 100 ms. While no objective data is available, Philips, reportedly in consultation with customers, has decided a 50-ms delay is the threshold. "When the delay is below 50 ms, nobody will be able to see it," Martin said. Meanwhile, iSuppli Corp.'s principal analyst, Shyam Nagrani, cited a far lower number. "It can be a big problem if the deinterlacing delay is greater than one field, i.e., 16.6 ms," Shyam said. "Different deinterlacers use two, three, four or five fields. Most of the deinterlacers use only two fields. It means that the delay is little more than one field." Micronas will introduce two devices, both leveraging on-chip RAM to delay audio information to keep lip movements in synch. One is a programmable delay function integrated onto the company's Multistandard Sound Processor family. The other, a standalone audio delay chip, integrates a controller and RAM. Micronas targets the fast-growing, mainstream flat-panel TV market segment with its Multistandard Sound Processor chip, called MSP 44/46xyK, featuring the new on-chip delay functionality. Delay time can be programmed to up to 80 ms (mono) or 40 ms (stereo), the company said. If longer delays are needed, Micronas' Hepp said, OEMs can use a new standalone audio delay IC, called MAD 4868A. It can function as a companion chip to Micronas' sound processor or anyone's audio DSP. The configurable delay time in such a setup is between 680 ms (mono) and 85 ms (eight channels). To optimize cost, there is no active component on the MAD 4868A, Hepp said. Alternatively, OEMs can address lip sync with separate chips - an external RAM and a RAM controller. "That would require as many as 40 connecting wires to connect the two. There are also significant EMC issues," Hepp said. Both chips are available in samples today with volumes scheduled for early summer. While Micronas has taken the on-chip RAM approach to delay the audio and solve the lip-synching problem, there are different approaches. Philips, for one, "fixed the problem," said Martin, via a system-level design by tweaking video enhancement software to minimize delay while adding a new feature - which automatically delays audio to match video - to its high-end Nexperia Home Entertainment Engine: the PNX8550. The IC, unveiled last year, is a system-on-chip TV that supports all digital and analog TV standards; CRT, LCD and plasma displays; 100-Hz progressive-scan video; high-definition video decode and display and many picture-enhancing features. Because audio/video processing is on the same chip, sharing the same memory, Philips did not have to integrate more memory for audio delay, Martin said. "Although we haven't heavily advertised it, lip-synching is one of the features on PNX8550." Digital TVs using PNX8550 are expected to reach the consumer market later this year. Meanwhile, there's an even simpler solution, said iSuppli's Shyam. That is "to use two-field deinterlacing where the video quality is almost as good as three- or four-field processing." Most customers could not tell the difference between two- or three-field processing, Shyam said. "So in the interests of costs and complexity we stuck with two-field deinterlacing." Copyright =A9 2003 CMP Media ---------------------------------------------------------------------- You can UNSUBSCRIBE from the OpenDTV list in two ways: - Using the UNSUBSCRIBE command in your user configuration settings at FreeLists.org - By sending a message to: opendtv-request@xxxxxxxxxxxxx with the word unsubscribe in the subject line.