[visionegg] Re: slight lag in quicktime StartMovie on WXP

  • From: John Herrington <john.herrington@xxxxxxxx>
  • To: visionegg@xxxxxxxxxxxxx
  • Date: Mon, 21 May 2007 17:15:44 -0400

A couple weeks back Andrew gave me some suggestions about how I might handle a time lag I was noticing when starting up a movie file via visionegg's quicktime functions (running on a PC). His advice was right on the money - I was able to write the program to introduce the lag up front in my program, before actually presenting the video. The key seemed to be starting the movie and running one swap_buffers() command to induce the file loading, then "rewinding" the video again right before playing it for real. One doesn't have to actually run the draw command to get the loading to happen, so nothing appears on the screen. I found that it didn't matter whether I loaded the first frame of a video, or the first 10 or 100 frames - in each case, the lag was eliminated when getting to the part of the program that actually plays the video.


Below I have pasted a version of the quicktime.py demo program providing extra code to do the preloading. The new code I added is sectioned off with comments. I arrived at the new code quite empirically, playing around with the visionegg qt commands until I got something that worked. Which is to say that there may be better/more elegant solutions to this problem.

One thing to note, FYI - I've found that the lag I initially observed (see beginning of thread) happens for some video files and not others. I'm very much a novice when it comes to understanding codecs and containers... it seems that the visionegg qt functions (and QTMLclient.dll) are able to handle some non-quicktime codecs, and it could be that these are the ones that have the lag in the first place. For example, the visionegg demo movie (water.mov) doesn't have a noticable delay upon running the quicktime.py program, so the fix isn't really needed there in the first place. But other video files I got from other sources (and possibly different codecs or codec parameters?) did indeed have a delay.

At any rate, many thanks for your suggestions - I am very happy with how visionegg is working with my video experiments!

John

#!/usr/bin/env python
"""Display a quicktime movie in the Vision Egg.

See also mpeg.py, which plays MPEG movies.
"""

import os, sys
import VisionEgg
VisionEgg.start_default_logging(); VisionEgg.watch_exceptions()

from VisionEgg.Core import *
from VisionEgg.Text import *
from VisionEgg.Textures import *
from VisionEgg.QuickTime import new_movie_from_filename, MovieTexture
from pygame.locals import *

screen = get_default_screen()
screen.set(bgcolor=(0,0,0))

if len(sys.argv) > 1:
    filename = sys.argv[1]
else:
filename = os.path.join(VisionEgg.config.VISIONEGG_SYSTEM_DIR,"data","water.mov")

movie = new_movie_from_filename(filename)
bounds = movie.GetMovieBox()
height = bounds.bottom-bounds.top
width = bounds.right-bounds.left

scale_x = screen.size[0]/float(width)
scale_y = screen.size[1]/float(height)
scale = min(scale_x,scale_y) # maintain aspect ratio

movie_texture = MovieTexture(movie=movie)

stimulus = TextureStimulus(
    texture=movie_texture,
    position = (screen.size[0]/2.0,screen.size[1]/2.0),
    anchor = 'center',
    mipmaps_enabled = False, # can't do mipmaps with QuickTime movies
    shrink_texture_ok = True,
    size = (width*scale, height*scale),
    )

text = Text( text = "Vision Egg QuickTime movie demo - Press any key to quit",
             position = (screen.size[0]/2,screen.size[1]),
             anchor = 'top',
             color = (1.0, 1.0, 1.0),
             )

viewport = Viewport(screen=screen,
                    stimuli=[stimulus, text])

##################### BEGIN NEW CODE #########################

#Load one frame of movie
movie.MoviesTask(0)
screen.clear()
swap_buffers()

#NOTE: at this point in the program there will be some loading delay for some
# types of videos.  This can be handled in an experiment by preloading all
# videos before actually starting the experiment (e.g., by indroducing some
# delay, or making the next part of the program begin via experiment keypress).
###################### END NEW CODE ##########################

movie.StartMovie()
frame_timer = FrameTimer()

##################### BEGIN NEW CODE #########################

#Rewind movie
movie.GoToBeginningOfMovie()

###################### END NEW CODE ##########################

quit_now = 0
while not quit_now:
    for event in pygame.event.get():
        if event.type in (QUIT,KEYDOWN,MOUSEBUTTONDOWN):
            quit_now = 1
    movie.MoviesTask(0)
    screen.clear()
    viewport.draw()

    swap_buffers() # display the frame we've drawn in back buffer
    frame_timer.tick()
    if movie.IsMovieDone():
        movie.GoToBeginningOfMovie()

frame_timer.print_histogram()


Andrew Straw wrote:
Dear John,

I don't know if there are any QuickTime functions that would help with what you want, but given the behavior you describe, the following might help:

Render the movie once (or at least the first few frames) without drawing it to the screen. Then issue a movie.GoToBeginningOfMovie() command and start your main loop that does display the movie there. To render without drawing to the screen, you have 2 options immediately apparent: 1) draw frames as in the demo quicktime.py, but simply skip the swap_buffers() command. 2) Call the MovieTexture instance's update() method yourself while the MovieTexture isn't being rendered to the screen.

Option 1 is conceptually simpler, but it less elegant, as it does more work (actually completely drawing the movie to the back-buffer, just never flipping to that back buffer). Option 2 is a little trickier, because it requires you to set up some of the infrastructure necessary to call the MovieTexture's update method(). But I think both methods should work. And both, I think, should also work to pre-load one movie while another is playing. For option 2, this is probably readily obvious, but for option 1, after drawing the un-displayed movie to the back buffer, issue a screen.clear() command, draw your "real" scene, and then finally issue a swap_buffers() call.

(FWIW: I'm afraid that, despite writing the VE's QuickTime stuff (and one or two other things in QuickTime), I know relatively little regarding the QuickTime API. I generally find Apple's documentation and terminology so Apple-centric (and often ObjC centric) that there's a significant effort involved just finding and comprehending what they have written. In this case, QuickTime has extensive documentation, but I have yet to really understand even the fundamentals: http://developer.apple.com/reference/QuickTime/index.html . I do recall being an option when saving quicktime movies (With QuickTime Pro?) for "quick pre-loading" or something. Perhaps that's relevant.)

Anyhow, good luck, and I'd be happy to hear any reports.

Cheers!
Andrew

John Herrington wrote:
I'm using the bleeding edge version of visionegg (1.1.dev) to show quicktime movies as part of an eyetracking experiment. I'm running 1.1.dev on a Windows XP machine with an nvidia GPU. I've got a program up and running that works very well, with one caveat - there is a brief lag (perhaps 500 msec) in the experiment upon executing the StartMovie command. Upon rewinding and replaying the movie, there is no lag. The size of the lag appears to scale with the size of the quicktime file. I've observed this across a few PCs, all fairly modern/fast with nvidia GPUs. My hypothesis is that the lag is due to some loading or processing that happens as part of the StartMovie command.

My main question: is there some way to eliminate this lag? My thinking was that perhaps there is some way to take care of whatever preloading is related to the lag before actually displaying the movie. I took a look at qtlowlevel.py and it appears that this function wraps around QTMLClient.dll. I scoured the net for good documentation on this .dll, but could find very little there to guide me. It appears that the visionegg quicktime python files capitalize on a few specific functions that are part of QTMLClient.dll (such as StartMovie and GoToBeginningOfMovie). Are there other QTMLClient functions that might help in some sort of preloading?

I realize that the visionegg quicktime functions are bleeding edge... but if anyone can think of a solution to the lag issue (or for that matter, point me to whatever QTMLClient docs. were referenced when creating the qtlowlevel.py functions), that would be excellent. I'd be happy to send my experiment and corresponding video files if that would help diagnose the problem.

Thanks!

John Herrington, Ph.D.
Yale University

======================================
The Vision Egg mailing list
Archives: //www.freelists.org/archives/visionegg
Website: http://www.visionegg.org/mailinglist.html

======================================
The Vision Egg mailing list
Archives: //www.freelists.org/archives/visionegg
Website: http://www.visionegg.org/mailinglist.html

======================================
The Vision Egg mailing list
Archives: //www.freelists.org/archives/visionegg
Website: http://www.visionegg.org/mailinglist.html

Other related posts: