[kinovea-dev] Re: C# 3.0 in Kinovea sources
- From: Joan <joan@xxxxxxxxxxx>
- To: kinovea-dev@xxxxxxxxxxxxx
- Date: Wed, 17 Aug 2011 18:08:02 +0200
Thinking about it again, this is probably the wrong level of abstraction.
The player screen should not have to know all this stuff about
buffering, etc. It just wants a frame to render.
There is also a need to support more file formats than what FFMpeg can
offer, specifically a reader for animated GIFs and a reader for a
sequence of individual images.
The abstraction thus needs to be a level higher. All file readers must
implement common services that other parts will use. (getting
thumbnails, moving to next frame, moving to a given spot, etc.)
The fact that the FFMpeg-backed reader will use a buffer is probably an
implementation detail of this particular reader. The animated GIF reader
may on the other hand load the entire content in memory righ away.
So, before diving into asynchronous decode, the abstraction layer for
file readers must be in place with a clean interface. Ideally the extra
readers would be loaded dynamically in a plug-in fashion.
I'll probably start a branch for this because it will break everything !
Le 16/08/2011 15:05, Joan a écrit :
> Regarding threading:
>
> Trying to organize thoughts I wrote a small page on the wiki with
> tentative system.
> http://www.kinovea.org/wiki/doku.php/codeasyncdecode
>
> It is a bit more complicated than what I wrote in the last mail
> because it also account for dropping in the decoding thread.
> If you have some time please review it and tear it appart :-)
> Especially scenario 2 which has some blurry zone. When the decoding is
> not ready at the right time, should we present the frame later on if
> we don't have anything more recent, or drop it anyway ?
> Thanks
>
>
>
>
> Le 16/08/2011 12:28, Joan a écrit :
>> Re: Tracking:
>> There is indeed some improvements possible to make the tracking more
>> robust in time.
>> The main issue being that by the time we look for the object in frame
>> n, frame n+1 is still on disk so we can't really anticipate. (in the
>> general case where there is no caching and no buffering).
>>
>> The underlying pattern-matching algorithm is quite efficient at
>> finding the pattern in the current frame, but yes, when the object is
>> occluded it fails and there is currently no cure implemented.
>> (May change when there is async buffering…)
>>
>> For the track tool we drop tracking altogether when under a given
>> similarity threshold.
>> (We also drop when the user moves several frames at a time, simply
>> because we don't have the result for the immediately preceding frame.)
>>
>> For the more general "trackability" framework, the tracking manager
>> will have to store results in a sparse manner. There should be no
>> dropping, loosing accuracy is less critical than with the track tool
>> where it was better to resort to manual positioning. (could be
>> critical for some tools though)
>>
>> I've started looking at data structure to hold the list of
>> coordinates but couldn't find a built-in structure suitable so far.
>> Any ideas ?
>> (Specs: dictionary like to retrieve the coords at time t, but also
>> linked with preceding value in time to initialize the search window).
>> Or an entirely different approach ?
>>
>>
>>
>> Re: Blocking queue: Thanks for pointing this out!
>>
>> Unfortunately for most video formats, dropping can only happen for
>> the rendering part, not for decoding. We may want to drop the next
>> frame, but we still need to decode it in order to rebuild the frame
>> after that…
>>
>> For the rendering part, some sort of time based dropping is already
>> in place. (maybe it's a bit contrived :-))
>> 1 - The timer runs in a separate thread.
>> 2 - If we are still decoding or rendering the previous frame by the
>> time the tick happens, we increment the drop counter and we don't ask
>> for this frame (neither decode nor render).
>> 3 - On next tick, we directly ask for the right frame.
>>
>> In 3, the decoder still has to decode all the dropped frames before
>> serving the right one ! It's a shame but that's due to video
>> compression algorithms.
>> I think this approach is currently working only for spurious load on
>> the CPU. For a video that is constantly too demanding, it doesn't
>> work. (automatic slow down is then triggered if the drop counter is
>> too high)
>>
>> Ok. How could this span out with a buffered input ?
>>
>> 1 - Timer runs in a separate thread.
>> 2 - If we are not currently rendering, try to dequeue one frame.
>> 3 - If the queue is empty, increment the drop counter and go on, do
>> not block/wait.
>> 4 - On next tick, dequeue as many frames as were dropped to get to
>> the right one.
>> Blocking would only happen on enqueue, if full.
>>
>> 2b - If we are busy rendering or computing tracks, increment drop
>> counter as well and don't wait.
>>
>> Does it sound ok ?
>> It is not addressing videos where the decoding itself would be
>> unsustainable. For this scenario, caching would be the fallback.
>> joan.
>>
>>
>>
>>
>>
>>
>> Le 15/08/2011 05:59, Hugh a écrit :
>>>
>>> With tracking it is only ever one frame ahead? Does this not cause
>>> problems with things like sun flare? I would of through that if the
>>> tracking is having trouble (like say it can’t find a reasonable
>>> object, like self-reported “health” of 70%) then it would try on the
>>> next frame? Using the health from 2 frames constantly you might have
>>> a much better assurance of not losing the object to a fake. However
>>> I’ve never done any work in this area so I’m really guessing here.
>>>
>>> I was just thinking about the whole image blocking queue. While
>>> using a normal video at 25 frames a second it may be fine, when you
>>> start working with high speed cameras an issue could occur. In slow
>>> mode it is no problem, but when the play speed outperforms the
>>> system there is a few issues:
>>>
>>> -Decoder could not keep up.
>>>
>>> -Display system could not keep up.
>>>
>>> -Filters/Tracker could not keep up.
>>>
>>> In all these cases the video would slow down, which in my sport is a
>>> terrible outcome as I would rather frames are dropped unnoticed to
>>> have a better idea of timing. So the system needs to be able to drop
>>> frames at the area which is not keeping up. So if it cannot decode
>>> at speed, it needs to drop frames in time. If it cannot display them
>>> in time, it needs to skip them etc. So the queue needs to be time
>>> based (so the system knows it is displaying the right one at the
>>> right time, even if it missed some), not just taking them out in order.
>>>
>>> So coming out it must check time, and not display it if it is out of
>>> date (unless it is the last frame). The decoding must also work out
>>> if it is getting behind and skip a frame.
>>>
>>> WPF does sound good. I have not used it, but I like the effects that
>>> can be made and also the use to the ribbon. This is very good in
>>> windows movie maker and I can see its usefulness in software like
>>> there where it can lower screen usages by spread out tool bars (plus
>>> removing the toolbar). However this does sound like a bit of work.
>>>
>>> *From:*kinovea-dev-bounce@xxxxxxxxxxxxx
>>> [mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] *On Behalf Of *Joan
>>> *Sent:* Monday, 15 August 2011 5:15 AM
>>> *To:* kinovea-dev@xxxxxxxxxxxxx
>>> *Subject:* [kinovea-dev] Re: C# 3.0 in Kinovea sources
>>>
>>> On the topic of multithreading, I also wrote a small message on the
>>> forum the other day that lists what is currently multithreaded
>>> <http://www.kinovea.org/en/forum/viewtopic.php?id=385>.
>>>
>>> Trying to identify other stuff that would gain at running on several
>>> threads :
>>> 1. Decoding/Rendering pipeline - obviously.
>>> 2. Image filters. (autocontrast, autocolor, sharpen). These filters
>>> are applied on each image independently, could be parallelized
>>> (people are not complaining about perf though).
>>> 3. Tracking several objects at the same time.
>>>
>>> Regarding tracking, it must be done each frame at a time because we
>>> need position n to anticipate position n+1. So, when tracking a
>>> single object maybe it doesn't make much sense to do it
>>> asynchronously. However, when tracking several objects, there could
>>> be one thread per tracker…
>>> Another aspect of tracking will be trackable tools, like a magnifier
>>> that follows the object, or an angle tool that tracks its points. (I
>>> would like to have a generic helper class that would extend any tool
>>> with tracking capability).
>>>
>>> Maybe there would be a way to track the object in all frames at
>>> once, but the difficulty is that the user must be able to manually
>>> move the target at any point, and then we should start tracking this
>>> instead.
>>>
>>> If I think of other parts of the application I will add them here.
>>> joan.
>>>
>>>
>>> Le 12/08/2011 21:10, Joan a écrit :
>>>
>>> Well, yes maybe you're right :-)
>>> I think some browsers report the framework version in the user
>>> agent. I will try to reinstall Piwik or Google analytics on the web
>>> site and get some stats on visitors. If most people are on 3.5
>>> already, there's no point staying behind!
>>>
>>> I viewed the 4.0 framework mostly as a bridge to dynamic languages,
>>> but yes, it seems they added important classes for parallel
>>> computing. It is indeed a bit too much of a step though in my
>>> opinion, but things change fast so…
>>> As far as I understand LINQ is completely available in 3.5 (even in
>>> 2.0 with the bridge lib)… (But 4.0 has a parallel implementation)
>>>
>>> I haven't progressed in the Direct2D code. I'm starting to think
>>> it's not the right approach. WPF is right there, designed to solve
>>> exactly the same problem. However converting Kinovea to WPF is a bit
>>> frightening for me… I've never used WPF so I don't really know how
>>> big of a task it would be.
>>>
>>> On the other hand, I'm ready to decouple decoding from rendering in
>>> the player. Having one thread for decoding and one for rendering
>>> would give an important speed boost as you mentioned before.
>>>
>>> I think this could be done with a blocking queue (as in the Python
>>> Queue class).
>>> - The decoding thread would constantly fill the queue and block when
>>> it's full, waiting for someone to "get" an element.
>>> - The main thread would read from the queue, and block when it's
>>> empty, waiting for someone to "put" an element.
>>> - In the normal case, the queue would have some elements and both
>>> thread would be able to work separately without waiting.
>>> If you think of another architecture don't hesitate.
>>> Joan.
>>>
>>>
>>>
>>> Le 12/08/2011 05:55, Hugh a écrit :
>>>
>>> Would it not be better to simply move it all forwards to 3/3.5/4? I
>>> know the dependencies would have to be updated to use newer
>>> versions, but Vista comes with 3, Windows 7 comes with 3.5.
>>>
>>> 3.5 has some performance boosts, but the native LINQ technology is
>>> only in .net 4, which maybe is too much of a step? This would
>>> require most systems to download an update (but this is pushed
>>> through by windows update anyway).
>>>
>>> Have you done any changes with the direct draw version? I was
>>> unfortunately in an accident and ended up in hospital for a little
>>> while. So I have not had time to check. I did try Kinovea on a new
>>> system and realised it was not using anywhere near the performance
>>> ability of any core.
>>>
>>> *From:*kinovea-dev-bounce@xxxxxxxxxxxxx
>>> <mailto:kinovea-dev-bounce@xxxxxxxxxxxxx>
>>> [mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] *On Behalf Of *Joan
>>> *Sent:* Friday, 12 August 2011 7:36 AM
>>> *To:* kinovea-dev@xxxxxxxxxxxxx <mailto:kinovea-dev@xxxxxxxxxxxxx>
>>> *Subject:* [kinovea-dev] C# 3.0 in Kinovea sources
>>>
>>> For quite a few versions now, the project is compiled with the C#
>>> 3.0 compiler, while still targeting the framework 2.0.
>>> I don't think there was any part of the code that actually used C#
>>> 3.0 features though.
>>> It's always a bit hard to know which features are pure syntactic
>>> sugar that will compile directly or if assemblies from the framework
>>> 3.5 will be needed.
>>>
>>> I have just added LINQBridge <http://code.google.com/p/linqbridge/>
>>> as a dependency (BSD licence). It's only in sandbox code but will be
>>> comitted shortly.
>>>
>>> This small assembly (60KB) basically allows us to use the features
>>> of C# 3.0, like extensions methods, the Action<> and Func<>
>>> delegates signature, LINQ queries, λ expressions, etc. (still
>>> targeting .NET 2.0)
>>>
>>> It just struck me as odd that Kinovea was a pet project to
>>> experiment new stuff and technologies, and that it was stuck in a
>>> 2005 language ! (The move to Mercurial DVCS follows the same logic)
>>> joan.
>>>
>>> PS: I'm using both SharpDevelop 4.0 and VC# Express 2008 as IDE.
>>>
>>
>
Other related posts:
