16.Sep.2005
It all started with a hint from Richard that he’s already using OpenGL as the display system in his OpenME editor.
In the old days, even before the whole 3d gaming hype, I used to be very much interested in VR-like graphics & visualizations – experimenting a lot with OpenGL and DirectX (sigh). I immediately started to think how we could possibly use all this overblown gaming-oriented 3d-hardware to aid the process of video editing. Well, it seems that the possibilities are there, just waiting to be explored.
Diva will use OpenGL as the rendering engine. If you ever took care to look at the video renderers available for mplayer or xine, you might have noticed that they do support OpenGL. But we will do much more than that. OpenGL (and all the hardware acceleration) will be used not only for the display. It will be used for the actual compositing and colorspace handling.
What does it mean from the user’s point of view?
The are some caveats obviously. The biggest one the fact, that even though major brands (Nvidia, Ati, Matrox) provide decent OpenGL drivers for Linux, they differ greatly in terms of hardware features (extensions) implemented. Shamefully enough, OpenGL doesn’t provide any defined standard in terms of dealing with YUV data.
But we can overcome this by adjusting the level of acceleration depending on the capabilities present. If you happen to have Nvidia FX or better, it will use the fragment shader, the one you know pretty well if you play Doom 3. If not – we will fallback to multi-texture blending.
Apple already does that with a proprietary extension to Ati drivers they developed for Quartz (which uses OpenGL). But they support only one YUV format (4:2:2) and it strikes me we can do much more. Instead of trying to pixel-convert the textures, we can accurately simulate the RGB->YUV equation with multi-textural blending. This way we can fed the the YUV data directly to the GPU without occupying too much of the AGP bus.
Now, I’d be grateful if some OpenGL-magician contacted me to discuss that before I actually started implementing it.
In other news Eugenia did it again. My monthly bandwidth went “poof” yesterday. I have to say thanks (again) to rjw who suggested at some point to host all my demos outside of this site. People managed to generate 70GB of traffic on the Novell Forge in the last two weeks.
Powered by Mephisto with a micro theme mod
9 Comments
Hi,
the latest GTK/Cairo uses Glitz (or will use, I am not sure), that already uses OpenGL to do the rendering. Does implementing OpenGL stuff at the Application Level really provide a value, or relying on Cairo could work ? I know Cairo is about vector graphics, not sure whether this is what you want, but I definitely think that DIVA should not directly deal with OpenGL.. Maybe there should be some library in-between ?
Just ideas, you are the expert ;) Regards, Sami
I’m not sure I understand what you mean. To clarify:
Cairo is used at the application level to draw the timeline using vector graphics. This is the application Diva-level.
OpenGL itself will be used for rendering at the low GStreamer level. Diva is just an application. The underlying engine (Gdv) is written in C and no OpenGL is exposed to the Diva itself. Also, OpenGL works at the X-level, which is pretty low. Curently the video display is handled by X overlays, which is also pretty low level.
The timeline-cairo has nothing to do with the OpenGL-rendering. The only way to get all the OpenGL/hardware benefits is to work directly at the low X/OpenGL level.
This sounds very interesting. Have you considered what could be done with more interesting shaders, implementing realtime video filters… both static, like any of the GIMP’s filters, and dynamic, such as ripple/wave effects. Imagine realtime toon effects or animated impressionist paintings! More practically, noise reduction and colour enhancement would no doubt be very popular.
The shaders could be emulated in slower time on the CPU of machines whose graphics cards were not up to the task. Such a system would also lend itself nicely to use on a render farm.
Will this be exposed as part of your Gdv library? I realise I’ve been thinking in far too generalised terms here, but it would be nice to allow for reuse in other apps. I’d imagine that a generalised OpenGL filter processor and compositor could also be used in even the most high-end of video apps. Or maybe someone would use it in a video game if they could integrate the OpenGL surface into the 3d engine. Maybe even video chat?
No worries ;-)
Good to see you are making progress.
I really think that there needs to be an image processing library layered on top of OpenGL. This way all the many SVG libraries, video editors, gimps and kritas etc can use the same filters, and not have to reinvent how to do it - it can include acceleration for OpenGL, and also fallbacks, SMP blah blah, etc, etc. Basically, lots of people want to do imagy things with GPUs now, without abandonment of software only stuff, so band together! Pretty much like Core image on OS X.
Luxor, perchance. It can include strong optional cairo integration ( as a lot of people are going to want to be blending cairo surfaces together.)
To answer Sami Dalouche:
Glitz does provide an opengl backend for cairo. This does allow an application that draw with cairo to take advantage of OpenGL rendering.
However, no toolkits such as GTK plan on using OpenGL rendering. I believe the largest problem is due to locking in OpenGL drivers. Try running two windows of glxgears and you’ll see them each occassionally pause while the other moves.
The longterm plan for the linux desktop is to have cairo apps use the Xrender backend protocol to talk to an OpenGL X server such as Xegl.
OpenGL can certainly be used in the way you describe to do RGB YUV conversion using multi-texturing. I would be amazed if there doesn’t already exist some image processing OpenGL library that one can start hacking from already though. It seems like a problem that will have been solved multiple times.
I’ve got some code to stream YUV 4:2:0 frames from ffmpeg to an OpenGL textured quad, if you’re interested, Michael. Had to do it to get realtime HD playback of JPEGs from my laptop (ffmpeg’s YUV->RGB conversion seems a bit on the slow side).
www.coralcache.org
Hi Dan,
You mentioned that you have some code to stream YUV 4:2:0 frames from ffmpeg to an OpenGl textured quad. I’m quite interested with it. Can you share the information?
Thanks.
Sorry, comments are closed for this article.