[kinovea-dev] Re: Direct2D

  • From: Hugh <greatview@xxxxxxxxx>
  • To: <kinovea-dev@xxxxxxxxxxxxx>
  • Date: Thu, 28 Apr 2011 11:36:21 +1000

Thanks for the details.


I just recompiled it with VC++ 2008 to 3.5 (because I?m using C# 2010 with
3.5).  This fixed the errors.



From: kinovea-dev-bounce@xxxxxxxxxxxxx
[mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] On Behalf Of Joan
Sent: Monday, 25 April 2011 8:05 PM
To: kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] Re: Direct2D


Last time I had this error it was because the ffmpeg dll (avcodec-52.dll,
avformat-52.dll, etc) weren't properly copied to the bin directory.
The PlayerServer assembly couldn't find them and was throwing this very
error at the first time we tried to load it.
(In the IDE, under the Root project, you will see these dll listed as simple
files with no build action other than "copy to the output directory", this
was done to prevent this issue).
So, check that these files are indeed in the output directory

Be sure to force the platform to x86 (instead of Any CPU). (If I recall
correctly that was a limitation of the C++ Express IDE where we couldn't
change the option)

Also, the VC++ Express 2010 IDE cannot easily be made to target the .Net 2.0
(or 3.5), it only works with 4.0 which is a totally different beast, maybe
that is causing an issue.
The VC++Express 2008 can still be downloaded here

If you are using the full Visual Studio, that shouldn't be an issue though.
But when you rebuild it, make sure the target framework is 2.0 (or 3.5). I'm
not sure if this info is versionned.

The C# projects in the Direct2D branch and everywhere still target the 2.0
by the way. Should be enough.
Rebuild the C++/CLI project, then rebuild and launch the C# solution, do you
still get the error ? 

Le 25/04/2011 03:39, Hugh a écrit : 

I?ve been trying to build this branch however I am coming into an error:



System.IO.FileLoadException was unhandled

  Message=Could not load file or assembly 'PlayerServer,
Version=, Culture=neutral, PublicKeyToken=null' or one of its
dependencies. The application has failed to start because its side-by-side
configuration is incorrect. Please see the application event log or use the
command-line sxstrace.exe tool for more detail. (Exception from HRESULT:


  FileName=PlayerServer, Version=, Culture=neutral,



       at Kinovea.ScreenManager.ScreenManagerKernel..ctor()

       at Kinovea.Root.RootKernel.BuildSubTree() in
D:\Programming\Kinovea\Root\Kernel.cs:line 180

       at Kinovea.Root.RootKernel..ctor() in
D:\Programming\Kinovea\Root\Kernel.cs:line 133

       at Kinovea.Root.Program.Main() in
D:\Programming\Kinovea\Root\Program.cs:line 69

       at System.AppDomain._nExecuteAssembly(Assembly assembly, String[]

       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence
assemblySecurity, String[] args)

       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()

       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)

       at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)

       at System.Threading.ThreadHelper.ThreadStart()

  InnerException: System.Runtime.InteropServices.COMException

       Message=The application has failed to start because its side-by-side
configuration is incorrect. Please see the application event log or use the
command-line sxstrace.exe tool for more detail. (Exception from HRESULT:





I?m using C#2010 with the projects set to 3.5 like before.  When I use the
old playerserver.dll it will load (however of course it is missing the
required features for playing the video), so something in the next
configuration is not working.  Would you have an idea what is missing?  (I?m
not a .net programmer normally ;)).



From: kinovea-dev-bounce@xxxxxxxxxxxxx
[mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] On Behalf Of Joan
Sent: Tuesday, 22 March 2011 5:28 AM
To: kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] Re: Direct2D


Yes, the reverse operation. After the frame is decoded by FFmpeg, it now
goes straight to a Direct2D bitmap. But I couldn't find an easy way to get a
regular GDI Bitmap out from it, to apply image filters for example.
I think one have to go through something called WIC or something, but that
is not implemented in SlimDX yet.
Maybe there's another solution? Or maybe asking FFmpeg again and getting a
GDI+ Bitmap just for the operation? 
If you can think up of something?

Le 21/03/2011 18:47, Hugh a écrit : 



By getting a GDI image, do you mean the reverse of this temporary conversion
function that I was using before?


These days extra memory like this is not an issue, and if in the future the
whole process is faster, the buffer does not need to cover so much.  So I
think this is not an issue.





From: kinovea-dev-bounce@xxxxxxxxxxxxx
[mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] On Behalf Of Joan
Sent: Saturday, 19 March 2011 10:08 PM
To: kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] Re: Direct2D



Update on the Direct2D front.

First I made some changes to the trunk to keep the diff with the branch to a

The most important is decoding directly to BGRA pixel format (r406 and

This means a 25% increase in memory requirements (32bpp against 24bpp), so
for similar memory settings in preferences, the working zone will be 25%
shorter in duration.
On the other hand, it gives about 10% speed boost on the rendering part,
depending on image size. (because it's the .NET internal format so an extra
copy is avoided).
This should be most visible for big image size when images have been
extracted to memory already.

Then I went on to clean up the Direct2D sandbox code I had to create the
- After decoding, the image is stored by the Renderer itself (in its
preferred format to avoid any further conversion). 
- Then we call Render() which use the stored image.

However, several problems:
1. There is a flickering issue.
2. I don't know how to get back a GDI Bitmap from the Direct2D code.
There 31 instances of code where this is needed (to create the key image
thumbnail, to save to disk, etc.)

Branch is created here :

Check the new folder ScreenManager/PlayerScreen/Rendering/?

Some experiments to do :

0. Switch Renderers in FrameServerPlayer constructor.

1. Painting on the Panel or on the PictureBox ?
The control to paint on is passed to m_FrameServer.Renderer.Initialize() in
the PlayerScreenUserInterface constructor.
Then a call to .Render() must be done depending on the paint invocation
method (see below).

2. Calling the rendering directly or through Control.Invalidate() ? (See

3. in GDI+, Getting the Graphics object from the Paint event or from the
handle of the control we draw on (flickering issue ?).

Note that tests must be done without triggering the memory caching of
frames. (use long video or low values in preferences)
Any function needing a Bitmap copy of the current frame will likely crash
for now :-) (key images, saving, etc.)

Let me know if you try some experiments or if anything seems unclear or


Le 10/03/2011 10:40, Hugh a écrit : 

Cool!  Sounds like you are making some serious in roads to getting it
working smoothly.


If the images are held in the render I think it is ok, as long as the
different display classes can tell it how to treat this image.  I think the
best is when they are all rendered in one single output controller and it is
fed a list of classes to go through.  Then the images can more easily be
rendered on top of each other, rotated around etc.  Sometime a class could
be made for image morphing, and then it would be the one passed to this
output controller.  When it is called, it can then hold the child classes
and be an intermediate that does the work.


I hope you can understand what I mean.


I can?t wait to see the code in the tree so I can mess a little with it ;)
However time is still my enemy for the next few weeks.


Good work with the progress!  For me this is the biggest thinking holding
the program back, as I really would love to be able to play videos from the
cameras I use directly.  But as they get higher quality, the program needs
to keep up.



From: kinovea-dev-bounce@xxxxxxxxxxxxx
[mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] On Behalf Of Joan
Sent: Sunday, 27 February 2011 9:17 PM
To: kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] Re: Direct2D


Follow up?

I managed to remove the temporary Bitmap conversion :-) 
In the end it was mostly a matter of finding the right pixel format to go

For some reason, I had always been converting from FFmpeg to a Bitmap in
Format24bppRgb, which is not optimal since the internal Bitmap format seems
to be Format32bppPArgb.
So the first change was to convert from the internal FFmpeg format (YUV420P)
to BGR32 (the naming of the Drawing.Bitmap enum is reversed) instead of
This will also positively impacts the GDI+ code.

The D2D code is expecting data in a similar format than Drawing.Bitmap (here
it's called DXGI_FORMAT_B8G8R8A8_UNORM) - so I could use the existing
snippet for Bitmap conversion, providing the raw data from the FFmpeg

The architecture is still a bit tentative (now it's the Renderer that holds
the image to be drawn) 

I'm having rendering issues though, there are black frames once in a while.
I don't know where this is coming from.
(I went with the asynchronous .Invalidate() call on the background panel for
these tests.)

Anyway, I'll create a branch next week and push what I have so far.

Le 22/02/2011 14:20, Hugh a écrit : 

Cool.  Sorry for the late replay, I?ve been crazy busy recently.  I was
wondering if you had any luck getting the decoding straight onto the directX
style bitmap?



From:  <mailto:kinovea-dev-bounce@xxxxxxxxxxxxx>
kinovea-dev-bounce@xxxxxxxxxxxxx [ <mailto:kinovea-dev-bounce@xxxxxxxxxxxxx>
mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] On Behalf Of Joan
Sent: Wednesday, 16 February 2011 6:25 PM
To:  <mailto:kinovea-dev@xxxxxxxxxxxxx> kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] Re: Direct2D


Just a quick note to say that I'll try to focus on the Direct2D rendering
stuff for a while.
Hopefully, to get some sort of a working base that we can build upon and
refactor later?

Le 14/01/2011 23:42, Hugh a écrit : 

I?m in a rush, so just quickly:


As long as there is a sound way to destroy it and any threads it may create,
then all is good!  I just came from outside C# so not so use to it all.

The DrawVideoBitmap() was very much a hack to try and get it to work in the
current program.  It would be good if it can draw straight onto a DirectX
style object, and do all rendering on it (there are many functions for it).
Obviously the more it is in the hardware (and not just copied over), the
best results will occur.

I don?t know enough about FFMPEG to know if it is possible to draw from it
onto these surfaces, but I?m sure it must be since VLC is very quick at

The whole class was designed to have a module feel to it, but also to fit
into the current program.  I did something close to what you have to do the
rendering, but I hope the class can easily be changed to fit any new
designs.  As I did more work on it, I realised there were other bottle necks
in the program that would need to be fixed get it fully working (like
drawing onto DirectX objects, new rendering process etc).

I left the other classes (like DirectX 9 and 10) mainly because Windows XP
does not support Direct2D.  However I guess there is a time people must
decide to move on.  They would be much harder to work with as you must draw
the shapes in the perspective, not simply simple objects.  Direct2D also has
many other small advantages like for remote desktop, integration etc.  I
never tested them, but didn?t want to remove the options in case someone
wanted to go down this path in the future.  Personally I favour Direct2D the


At the moment I am rather limited for time (sorry!), but this is something I
would love to see in the program.





From:  <mailto:kinovea-dev-bounce@xxxxxxxxxxxxx>
kinovea-dev-bounce@xxxxxxxxxxxxx [ <mailto:kinovea-dev-bounce@xxxxxxxxxxxxx>
mailto:kinovea-dev-bounce@xxxxxxxxxxxxx] On Behalf Of Joan
Sent: Thursday, 13 January 2011 10:19 PM
To:  <mailto:kinovea-dev@xxxxxxxxxxxxx> kinovea-dev@xxxxxxxxxxxxx
Subject: [kinovea-dev] Re: Direct2D



I have tried the Direct2D / SlimDX code you sent.

I wasn't too sure as to how exactly use the control, here is what I did:
- Added a Direct2DControl member to PlayerScreenUserInterface.
- In the PlayerScreenUserInterface constructor I call
Direct2DControl.SetControl() passing in the center panel (to test on the
back panel for now)
- immediately call Direct2DControl.Run() to trigger initializations.
- At each new frame, (in PlayerScreenUserInterface.ShowNextFrame), I call
Direct2DControl.DrawVideoBitmap() passing in the decoded image,
- and Direct2DControl.Update() to actually draw on the panel.

Let me know if I did anything wrong. Did you also modify
PlayerScreenUserInterface in this way for your tests ?

It seems to work pretty well.
I've tried to instrument and get some figures regarding speed, but it proved
harder than I thought to really pinpoint the rendering time.
(Also I've recently changed machine and it's like everything is running
super fast :-))

The part taking the most time (about 10 to 20ms each frame on my machine)
would seem to be DrawVideoBitmap(). 
This function involves copying the whole GDI+ bitmap into a Direct2D bitmap
for later use. This is a step we can probably avoid completely by decoding
directly into a Direct2D bitmap from the FFMpeg code. 

This is maybe what you suggested in the forum thread... In the case of
pre-extraction to memory, we can also probably keep an in-memory array of
Direct2D bitmaps instead of an array of GDI+ bitmaps. I'm sure we'll find a
way to avoid this extra copy.

About your concern to release resources, I don't think we need to use a
separate form for rendering and react to the OnClose() event of it.
Screens must implement a BeforeClose() method (see AbstractScreen) which is
triggered when the screen is closed (close button, changing screens
configuration, etc.)
We should be able to use it to ask the PlayerScreen and the renderer to
dispose any outstanding resources.

We'll probably need to transform and simplify the DirectXInterface class.
Also, I don't know if we need the other classes, (Direct3D10Context, etc.) ?
If not, we might even merge the two for simplicity.

I haven't looked at the direct2D drawing primitives and how we might
implement the drawing tools visuals. 
The architecture discussion will probably depend on how these can be
abstracted between Direct2D and GDI+.

I will create a topic on the wiki to push some schematics of possible
architectures so we are clearer about what we discuss.
Let me know of your experiments.


Le 10/01/2011 21:34, Joan a écrit : 


I am super sorry for the delay.
Unfortunately, I haven't had much time to look into this, but I will.
(I have some issues at work with the company wanting to close the local
office, this has somewhat prevented proper focus on the project, but it's
going to resolve one way or the other in the next few weeks.)


Le 05/01/2011 17:34, Hugh a écrit : 

Hey Jean,


Was just wondering how the Direct2D code was working for you?  Did you have
any luck moving the drawing onto a new form?  Just didn?t see any recent
changes in the code and was hoping it is working!











Other related posts: