[kinovea-dev] Re: C# 3.0 in Kinovea sources

  • From: Joan <joan@xxxxxxxxxxx>
  • To: kinovea-dev@xxxxxxxxxxxxx
  • Date: Wed, 24 Aug 2011 17:12:01 +0200


Regarding locking, it will only lock during the actual action of pushing (or reading) a Bitmap to (from) the cache. So that should be a fairly rapid operation, since it just involves references, not copying actual data. Hmm, wait, the problem is probably that while rendering, the decoder shouldn't mess with the frame underlying data...

At the moment Kinovea is a fat gif reader ! :-) (Lives in the "FileTypes" branch). Now working on adapting the FFMpeg backend to conform to the VideoReader base class. Doing it in a synchronous way for a start, but with the dynamic cache. (btw, I toyed a bit more with Google Docs <https://docs.google.com/drawings/d/1eCi9w8_OxqtwwF8WVG5YSuPBM1iQvn4K5t1SMzDeku0/edit?hl=en_US> the other day :-))

There are some subtleties arising.
- When saving, we need the decoding to be synchronous all the way. Get a frame, encode, repeat. No drops. - When reading analysis data embedded in the file or in a companion file (.kva), we jump and decode the key images to initialize the thumbnails. Corrupts the cache with non contiguous frames. - When rolling over the end of the video. We now have two segments crossed. But doable.

My current issue is storing and freeing the actual frame data, even if synchronous. The cache exposes Bitmaps, and should be agnostic to the underlying decoder.

The .NET Bitmap is just a wrapper around the low level data buffer, and it is possible to create a Bitmap directly from the buffer that FFMpeg creates. This is cool because it avoids doing a wasteful copy. So as long as the C++ code keeps the buffer alive, the Bitmap is valid and can be used from C#.

Now instead of just one buffer for the current frame, we must keep all the buffers of the frames that have been pushed to the dynamic cache.
And we should free them when the frame is removed from the cache.
In the cache, calling .Dispose() on the Bitmap to forget will do just that, but the C++ code is not alerted, so we now have a dangling pointer...

Maybe having a flag on readers to say if the reader is the owner of the underlying data or not. (the Gif reader is not, as it decodes and pushes everything to the cache directly) (For the full working zone caching, a deep copy of the decoded frames was made, this may might change if a no-copy solution is possible...)
Cheers,
Joan.



Le 21/08/2011 11:36, Hugh a écrit :

There would be advantages either way. My worry would just be if you end up double coding things and a complication from too much in the one control. However there could be performance gains mixed together.

Just the decoder will then be the threading and locking will have to be mixed through. With separate classes it could be split apart. You want to be able to take images out of the cache while decoding more into it (don't want to have to wait for a whole group of frames to decode before pulling one out).

*From:*kinovea-dev-bounce@xxxxxxxxxxxxx [mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] *On Behalf Of *Joan
*Sent:* Friday, 19 August 2011 8:48 PM
*To:* kinovea-dev@xxxxxxxxxxxxx
*Subject:* [kinovea-dev] Re: C# 3.0 in Kinovea sources

I'm working in parallel on the Gif plugin and I think it helps for defining the abstraction level. I would like to simplify things as much as possible as viewed from the player, and hide implementation details.

One thing I think I was viewing differently than in your design, is who interacts with the Buffer. Instead of having the Buffer as a middle man between the Decoder and the Renderer, It could be an implementation detail of the Decoder.

Super high level overview <https://docs.google.com/drawings/d/17_2MmbrbaxTB4905eSycmFM-AAtWkbsXLlANbSYNsd0/edit?hl=en_US> of (how I understand) the two alternatives.

Here is an alternative for decoders. It more resembles an enumerator, with a "Current" property (I admit the existing design is also like an enumerator so it will ease the refactoring :-)).

{
    Open(path);
    Close();
    GetThumbs();

    MoveNext();
    MoveTo(timestamp);

    VideoFrame Current { get; }
}

These would be the basic primitives for decoders, and would be called directly from the rendering/controller thread. Move* methods would set the "Current" reference to point to a valid frame. (not showing Cache management, which is another beast that can't be completely hidden from player). Some higher level methods like MovePrev(), MoveFirst(), MoveBy(frames), can also be implemented in the base class for decoders and mapped on implementer's MoveTo(timestamp).

Now where the Buffer would fit into this ? It would just help implement MoveNext(). The background thread would start immediately at Open() and fill buffer continuously.

- MoveNext() would be implemented to return super fast:
If the frame has been decoded by the background thread and is in the buffer, just update ref to "Current". If not we will call it a drop and keep working with whatever is in "Current".

- MoveTo() would check if the target frame is in the buffer, if not: perform the seek, update "Current", and then return.
It doesn't have to be as fast as MoveNext, no concept of drops.

You are very right with keeping some of the previous frames around. So I guess it's not a Queue after all. The behavior will be entirely defined by which frame gets unloaded first when buffer is full.

The buffer may end up being very much like the cache, just more dynamic (and thread safe). I guess the Cache could actually be implemented completely within the Buffer class, it would just contain all the frames from the working zone instead of a sub section around the current cursor. Will have to think more about this, that would simplify the architecture. It's probably independant from where the buffer actually fit in the system.
joan.


Le 19/08/2011 07:59, Hugh a écrit :

Would something like this work:

Notes:

location is always a position in the video

render calls buffer for image which gets it from itself and gets required items from decoder

render creates image in the background the when ready swaps them to the visual surface

render holds a list of tools to do changes to the image

classes can be overloaded (so different decoders or renders can be used)

public class Decoder

{

               // Construction, loading

               public Decoder();

public result LoadVideo(stream/location); // Loads video from stream or given location

public properties GetProperties(); // Returns a structure with video properties

public int SetProperty(name, value); // Set a property

               // Extraction

public frame[] GetFrames(start, [optional] end); // Returns frames in the range

public frame[] GetThumbs([optional] number = 1, [optional] start offset); // Returns thumb frames

public int FillBuffer(buffer, start, end); // Checks buffer and passes required frames to the buffer PutFrame (one at a time to put frame as async method as they are decoded, not a group like getframes)

}

public class Buffer

{

               // Construction, loading

               public Buffer();

public LoadDecoder(decoder); // Passes a decoder to use

public SetSize(ahead, [optional] behind = ahead); // Sets buffer size before and after location

               public result GetSize();

               // Updating

public int UpdateBuffer([optional] current location); // Checks buffer and fills up as required using the decoder (returns if update was performed)

public bool UpToDate(); // Passes true if buffer is full and to the right location

public result GetStatus(); // Passes number of empty frames ahead and behind

public SetLocation(location); // Sets location in the video without update

               public result GetLocation();

public Clear(); // Empties the buffer

               // Extraction

public frame GetFrame(location, [optional] update buffer = 1); // Gets the frame and moves buffer (will also need to get from decoder if missing)

public int PutFrame(location); // Lets decoder place a frame into the buffer at location

}

public class Render

{

               // Construction

               public Render();

public LoadBuffer(); // Passes a buffer to use

               public SetSurface(hwnd);

               // To screen

public bool SwapFrame(); // Places rendred frame onto surface (back buffer to screen). Checks frame is ready to be drawn and returns true on sucess

               // Load next frame

public bool Render(location); // Gets image from buffer and starts drawing onto it (done in background memory then when is finished SwapFrame can move it to the visable surface

               // Tools

public id AddTool(tool); // Adds a tool to render onto the frame (like drawing, filter etc)

public RemoveTool(id); // Removes a tool

               ...

}

*From:*kinovea-dev-bounce@xxxxxxxxxxxxx <mailto:kinovea-dev-bounce@xxxxxxxxxxxxx> [mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] *On Behalf Of *Joan
*Sent:* Thursday, 18 August 2011 2:08 AM
*To:* kinovea-dev@xxxxxxxxxxxxx <mailto:kinovea-dev@xxxxxxxxxxxxx>
*Subject:* [kinovea-dev] Re: C# 3.0 in Kinovea sources

Thinking about it again, this is probably the wrong level of abstraction.
The player screen should not have to know all this stuff about buffering, etc. It just wants a frame to render.

There is also a need to support more file formats than what FFMpeg can offer, specifically a reader for animated GIFs and a reader for a sequence of individual images. The abstraction thus needs to be a level higher. All file readers must implement common services that other parts will use. (getting thumbnails, moving to next frame, moving to a given spot, etc.)

The fact that the FFMpeg-backed reader will use a buffer is probably an implementation detail of this particular reader. The animated GIF reader may on the other hand load the entire content in memory righ away.

So, before diving into asynchronous decode, the abstraction layer for file readers must be in place with a clean interface. Ideally the extra readers would be loaded dynamically in a plug-in fashion.

I'll probably start a branch for this because it will break everything !





Le 16/08/2011 15:05, Joan a écrit :

Regarding threading:

Trying to organize thoughts I wrote a small page on the wiki with tentative system.
http://www.kinovea.org/wiki/doku.php/codeasyncdecode

It is a bit more complicated than what I wrote in the last mail because it also account for dropping in the decoding thread. If you have some time please review it and tear it appart :-) Especially scenario 2 which has some blurry zone. When the decoding is not ready at the right time, should we present the frame later on if we don't have anything more recent, or drop it anyway ?
Thanks






Other related posts: