[kinovea-dev] Re: C# 3.0 in Kinovea sources
- From: Hugh <greatview@xxxxxxxxx>
- To: <kinovea-dev@xxxxxxxxxxxxx>
- Date: Thu, 18 Aug 2011 13:20:51 +1000
Did you ever look at using http://directshownet.sourceforge.net/about.html,
http://blogs.msdn.com/b/mf/archive/2009/11/13/windows-media-foundation-team-
blog.aspx or http://mfnet.sourceforge.net/? Using these system you would
get access to the built in hardware decoding, and all codex installed on a
system. Maybe FFMpeg can be a fall back if the other systems don’t find
anything. If you use WPF maybe these other systems are easier? I don’t
really know much about WPF.
On Windows 7 native applications (like media players and live gallery) will
try media foundation then if that fails they go to direct show. The built
in codecs in 7 are part of the media foundation. FFMpeg support can always
be achieved through their direct show filters which means you can support
both without having to have both interfaces if need be.
The Windows 7 Media Foundation x264 playback uses only 2-4% with 1080P,
which is really great in my view. That’s why I like programs that use full
advantage of it. WebM is also available for the Media Foundation.
I think the buffer should be filled up when you load a video, for example it
could preload 2 or so seconds of video. That way it is ready to go when you
click play. Also it should keep a good amount after the current play
position (this is useful for going back a few frames so it does not have to
reload when stepping frames or moving backwards). Everything else on that
page looks good.
From: kinovea-dev-bounce@xxxxxxxxxxxxx [mailto:kinovea-dev-bounce@freelists.
org] On Behalf Of Joan
Sent: Thursday, 18 August 2011 2:08 AM
To: kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] Re: C# 3.0 in Kinovea sources
Thinking about it again, this is probably the wrong level of abstraction.
The player screen should not have to know all this stuff about buffering,
etc. It just wants a frame to render.
There is also a need to support more file formats than what FFMpeg can
offer, specifically a reader for animated GIFs and a reader for a sequence
of individual images.
The abstraction thus needs to be a level higher. All file readers must
implement common services that other parts will use. (getting thumbnails,
moving to next frame, moving to a given spot, etc.)
The fact that the FFMpeg-backed reader will use a buffer is probably an
implementation detail of this particular reader. The animated GIF reader may
on the other hand load the entire content in memory righ away.
So, before diving into asynchronous decode, the abstraction layer for file
readers must be in place with a clean interface. Ideally the extra readers
would be loaded dynamically in a plug-in fashion.
I'll probably start a branch for this because it will break everything !
Le 16/08/2011 15:05, Joan a écrit :
Regarding threading:
Trying to organize thoughts I wrote a small page on the wiki with tentative
system.
http://www.kinovea.org/wiki/doku.php/codeasyncdecode
It is a bit more complicated than what I wrote in the last mail because it
also account for dropping in the decoding thread.
If you have some time please review it and tear it appart :-) Especially
scenario 2 which has some blurry zone. When the decoding is not ready at the
right time, should we present the frame later on if we don't have anything
more recent, or drop it anyway ?
Thanks
Le 16/08/2011 12:28, Joan a écrit :
Re: Tracking:
There is indeed some improvements possible to make the tracking more robust
in time.
The main issue being that by the time we look for the object in frame n,
frame n+1 is still on disk so we can't really anticipate. (in the general
case where there is no caching and no buffering).
The underlying pattern-matching algorithm is quite efficient at finding the
pattern in the current frame, but yes, when the object is occluded it fails
and there is currently no cure implemented.
(May change when there is async buffering…)
For the track tool we drop tracking altogether when under a given similarity
threshold.
(We also drop when the user moves several frames at a time, simply because
we don't have the result for the immediately preceding frame.)
For the more general "trackability" framework, the tracking manager will
have to store results in a sparse manner. There should be no dropping,
loosing accuracy is less critical than with the track tool where it was
better to resort to manual positioning. (could be critical for some tools
though)
I've started looking at data structure to hold the list of coordinates but
couldn't find a built-in structure suitable so far. Any ideas ?
(Specs: dictionary like to retrieve the coords at time t, but also linked
with preceding value in time to initialize the search window).
Or an entirely different approach ?
Re: Blocking queue: Thanks for pointing this out!
Unfortunately for most video formats, dropping can only happen for the
rendering part, not for decoding. We may want to drop the next frame, but we
still need to decode it in order to rebuild the frame after that…
For the rendering part, some sort of time based dropping is already in
place. (maybe it's a bit contrived :-))
1 - The timer runs in a separate thread.
2 - If we are still decoding or rendering the previous frame by the time the
tick happens, we increment the drop counter and we don't ask for this frame
(neither decode nor render).
3 - On next tick, we directly ask for the right frame.
In 3, the decoder still has to decode all the dropped frames before serving
the right one ! It's a shame but that's due to video compression algorithms.
I think this approach is currently working only for spurious load on the
CPU. For a video that is constantly too demanding, it doesn't work.
(automatic slow down is then triggered if the drop counter is too high)
Ok. How could this span out with a buffered input ?
1 - Timer runs in a separate thread.
2 - If we are not currently rendering, try to dequeue one frame.
3 - If the queue is empty, increment the drop counter and go on, do not
block/wait.
4 - On next tick, dequeue as many frames as were dropped to get to the right
one.
Blocking would only happen on enqueue, if full.
2b - If we are busy rendering or computing tracks, increment drop counter as
well and don't wait.
Does it sound ok ?
It is not addressing videos where the decoding itself would be
unsustainable. For this scenario, caching would be the fallback.
joan.
Le 15/08/2011 05:59, Hugh a écrit :
With tracking it is only ever one frame ahead? Does this not cause problems
with things like sun flare? I would of through that if the tracking is
having trouble (like say it can’t find a reasonable object, like
self-reported “health” of 70%) then it would try on the next frame? Using
the health from 2 frames constantly you might have a much better assurance
of not losing the object to a fake. However I’ve never done any work in
this area so I’m really guessing here.
I was just thinking about the whole image blocking queue. While using a
normal video at 25 frames a second it may be fine, when you start working
with high speed cameras an issue could occur. In slow mode it is no
problem, but when the play speed outperforms the system there is a few
issues:
Decoder could not keep up.
Display system could not keep up.
Filters/Tracker could not keep up.
In all these cases the video would slow down, which in my sport is a
terrible outcome as I would rather frames are dropped unnoticed to have a
better idea of timing. So the system needs to be able to drop frames at the
area which is not keeping up. So if it cannot decode at speed, it needs to
drop frames in time. If it cannot display them in time, it needs to skip
them etc. So the queue needs to be time based (so the system knows it is
displaying the right one at the right time, even if it missed some), not
just taking them out in order.
So coming out it must check time, and not display it if it is out of date
(unless it is the last frame). The decoding must also work out if it is
getting behind and skip a frame.
WPF does sound good. I have not used it, but I like the effects that can be
made and also the use to the ribbon. This is very good in windows movie
maker and I can see its usefulness in software like there where it can lower
screen usages by spread out tool bars (plus removing the toolbar). However
this does sound like a bit of work.
From: kinovea-dev-bounce@xxxxxxxxxxxxx [mailto:kinovea-dev-bounce@freelists.
org] On Behalf Of Joan
Sent: Monday, 15 August 2011 5:15 AM
To: kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] Re: C# 3.0 in Kinovea sources
On the topic of multithreading, I also wrote a small message on the forum
the other day that lists what is currently multithreaded
<http://www.kinovea.org/en/forum/viewtopic.php?id=385> .
Trying to identify other stuff that would gain at running on several threads
:
1. Decoding/Rendering pipeline - obviously.
2. Image filters. (autocontrast, autocolor, sharpen). These filters are
applied on each image independently, could be parallelized (people are not
complaining about perf though).
3. Tracking several objects at the same time.
Regarding tracking, it must be done each frame at a time because we need
position n to anticipate position n+1. So, when tracking a single object
maybe it doesn't make much sense to do it asynchronously. However, when
tracking several objects, there could be one thread per tracker…
Another aspect of tracking will be trackable tools, like a magnifier that
follows the object, or an angle tool that tracks its points. (I would like
to have a generic helper class that would extend any tool with tracking
capability).
Maybe there would be a way to track the object in all frames at once, but
the difficulty is that the user must be able to manually move the target at
any point, and then we should start tracking this instead.
If I think of other parts of the application I will add them here.
joan.
Le 12/08/2011 21:10, Joan a écrit :
Well, yes maybe you're right :-)
I think some browsers report the framework version in the user agent. I will
try to reinstall Piwik or Google analytics on the web site and get some
stats on visitors. If most people are on 3.5 already, there's no point
staying behind!
I viewed the 4.0 framework mostly as a bridge to dynamic languages, but yes,
it seems they added important classes for parallel computing. It is indeed a
bit too much of a step though in my opinion, but things change fast so…
As far as I understand LINQ is completely available in 3.5 (even in 2.0 with
the bridge lib)… (But 4.0 has a parallel implementation)
I haven't progressed in the Direct2D code. I'm starting to think it's not
the right approach. WPF is right there, designed to solve exactly the same
problem. However converting Kinovea to WPF is a bit frightening for me…
I've never used WPF so I don't really know how big of a task it would be.
On the other hand, I'm ready to decouple decoding from rendering in the
player. Having one thread for decoding and one for rendering would give an
important speed boost as you mentioned before.
I think this could be done with a blocking queue (as in the Python Queue
class).
- The decoding thread would constantly fill the queue and block when it's
full, waiting for someone to "get" an element.
- The main thread would read from the queue, and block when it's empty,
waiting for someone to "put" an element.
- In the normal case, the queue would have some elements and both thread
would be able to work separately without waiting.
If you think of another architecture don't hesitate.
Joan.
Le 12/08/2011 05:55, Hugh a écrit :
Would it not be better to simply move it all forwards to 3/3.5/4? I know
the dependencies would have to be updated to use newer versions, but Vista
comes with 3, Windows 7 comes with 3.5.
3.5 has some performance boosts, but the native LINQ technology is only in .
net 4, which maybe is too much of a step? This would require most systems
to download an update (but this is pushed through by windows update anyway).
Have you done any changes with the direct draw version? I was unfortunately
in an accident and ended up in hospital for a little while. So I have not
had time to check. I did try Kinovea on a new system and realised it was
not using anywhere near the performance ability of any core.
From: kinovea-dev-bounce@xxxxxxxxxxxxx [mailto:kinovea-dev-bounce@freelists.
org] On Behalf Of Joan
Sent: Friday, 12 August 2011 7:36 AM
To: kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] C# 3.0 in Kinovea sources
For quite a few versions now, the project is compiled with the C# 3.0
compiler, while still targeting the framework 2.0.
I don't think there was any part of the code that actually used C# 3.0
features though.
It's always a bit hard to know which features are pure syntactic sugar that
will compile directly or if assemblies from the framework 3.5 will be
needed.
I have just added LINQBridge <http://code.google.com/p/linqbridge/> as a
dependency (BSD licence). It's only in sandbox code but will be comitted
shortly.
This small assembly (60KB) basically allows us to use the features of C#
3.0, like extensions methods, the Action<> and Func<> delegates signature,
LINQ queries, λ expressions, etc. (still targeting .NET 2.0)
It just struck me as odd that Kinovea was a pet project to experiment new
stuff and technologies, and that it was stuck in a 2005 language ! (The move
to Mercurial DVCS follows the same logic)
joan.
PS: I'm using both SharpDevelop 4.0 and VC# Express 2008 as IDE.
Other related posts:
