get from 30 to 60 fps for “free”
- Главная
- Разное
- Дизайн
- Бизнес и предпринимательство
- Аналитика
- Образование
- Развлечения
- Красота и здоровье
- Финансы
- Государство
- Путешествия
- Спорт
- Недвижимость
- Армия
- Графика
- Культурология
- Еда и кулинария
- Лингвистика
- Английский язык
- Астрономия
- Алгебра
- Биология
- География
- Детские презентации
- Информатика
- История
- Литература
- Маркетинг
- Математика
- Медицина
- Менеджмент
- Музыка
- МХК
- Немецкий язык
- ОБЖ
- Обществознание
- Окружающий мир
- Педагогика
- Русский язык
- Технология
- Физика
- Философия
- Химия
- Шаблоны, картинки для презентаций
- Экология
- Экономика
- Юриспруденция
SIGGRAPH 2010 презентация
Содержание
- 2. SIGGRAPH 2010 Dmitry Andreev (AND) dandreev@LucasArts.com
- 3. Agenda Stories, Ideas, Possibilities Excitement, Inspiration Better
- 4. Real-time DEMO
- 5. Frame Rate Up-conversion Natural Motion of Objects
- 6. Motion Estimation Block-matching Diamond Search Optical Flow
- 7. Video Frame Interpolation Build Inner frames from
- 8. Motion Vectors vs Velocities
- 9. No Need for Motion Estimation The Source
- 10. Running at 30 fps Game Extra 16.6
- 11. Running at 30 fps Game Extra 16.6
- 12. Motion Eye-tracking (60 fps) Observed @ 60Hz
- 13. Motion Eye-tracking (30 fps) Observed @ 60Hz
- 14. Motion Blur Helps to Remedy “Flickering”, but…
- 15. Motion Blur In Games
- 17. The Idea is … Render Velocity Buffer
- 18. The Idea is simple … Render Velocity Buffer Interpolate the Middle Frame
- 19. The Idea is very
- 20. Ideally (to be “free”) Reuse as Much
- 21. Typical Rendering Pipeline
- 22. Modified Rendering
- 23. One-frame Based Solution Pros No extra latency
- 24. INTERPOLATION
- 25. Very Basic Interpolation Pn-½ = f(
- 26. Original I
- 27. Interpolated Very basic interpolation II
- 28. Very Basic Interpolation III Based on Current Velocity
- 29. Camera Motions (TFU2) I 3rd Person Camera Translation
- 30. Camera Motions (TFU2) II 3rd Person Camera Translation Rotation
- 31. Artifact Minimization Static Geometry Velocity buffer filtering
- 32. Two-frame Based Solution Pros Smoother shadows, alpha,
- 33. Modified Velocity Rendering Re-render Characters At interpolated
- 34. Velocity Merging Very Simple and Efficient
- 35. Original I
- 36. Interpolated Velocity Merging II
- 37. Character Removal I Visual System’s Blind Spot Inspired
- 38. Character Removal I Visual System’s Blind Spot Inspired
- 39. Character Removal II Generate Mask
- 40. Character Removal II Generate Mask Leak Neighboring Image Patches
- 41. Character Removal II Generate Mask Leak Neighboring Image Patches Duplicate and offset Up
- 42. Character Removal II Generate Mask Leak Neighboring Image Patches Duplicate and offset Up Down
- 43. Character Removal II Generate Mask Leak Neighboring Image Patches Duplicate and offset Up Down Left
- 44. Character Removal II Generate Mask Leak Neighboring
- 45. Character Removal III Repeat N Times Blur
- 46. Character Removal IV
- 47. Improved Interpolation Based on Vmerged Sample half
- 48. Original I
- 49. Interpolated Velocity Merging and Character Removal II
- 50. Next Original In case of One-frame based Interpolation it is Unknown III
- 51. Typical Flipping Flip Normal Buffer
- 52. Modified Flipping Flip Interpolated
- 53. Custom flipping on XBox360 Bypass Direct3D Present
- 54. Real-time Implementation One-frame Based Interpolation of Dynamic
- 55. Motion Blur In TFU2 Current Solution Alpha
- 56. Future Work Transparency Customizations Re-render lightsabers, HUD
- 57. Acknowledgments Szymon Swistun
- 58. Words of Wisdom
- 59. Thank You
- 60. Thank You Questions? dandreev@LucasArts.com
Слайд 2
SIGGRAPH 2010
Dmitry Andreev (AND)
dandreev@LucasArts.com
Real-time Frame Rate Up-conversion
for Video Games
or how to
Слайд 3Agenda
Stories, Ideas, Possibilities
Excitement, Inspiration
Better Games
Happy Players
…
Technical Details are Simplified
Only current real-time
implementation is discussed
Слайд 5Frame Rate Up-conversion
Natural Motion of Objects
In Software
AVISynth + MSU FRC
WinDVD (Trimension)
In
Hardware
Display devices (120Hz HDTVs)
Display devices (120Hz HDTVs)
Слайд 6Motion Estimation
Block-matching
Diamond Search
Optical Flow
Phase Correlation
MAP/MRF
or take it from the
MPEG
stream :)
A
B
Motion Vectors
Слайд 7Video Frame Interpolation
Build Inner frames from outer frames
Using Motion Vectors
High-res estimation
is required
Scaling, rotation, transparency, dynamic lighting ???
Scaling, rotation, transparency, dynamic lighting ???
Слайд 9No Need for Motion Estimation
The Source of Motion is Known
Objects
Camera
Handle Special
Cases Differently
Transparency
Shadows, reflections, refractions
Characters ?
Transparency
Shadows, reflections, refractions
Characters ?
Слайд 10Running at 30 fps
Game
Extra 16.6 ms
More advanced rendering techniques
Deferred / Screen-space,
etc…
Less strict production process (easier to make)
Less strict production process (easier to make)
Слайд 11Running at 30 fps
Game
Extra 16.6 ms
More advanced rendering techniques
Deferred / Screen-space,
etc…
Less strict production process (easier to make)
User
Exposed to the same frame Twice
Less strict production process (easier to make)
User
Exposed to the same frame Twice
Слайд 12Motion Eye-tracking (60 fps)
Observed @ 60Hz
Display A, B, C, ...
Temporal Functions
Display
Eye
0
1/60
s
0
1/60 s
Слайд 13Motion Eye-tracking (30 fps)
Observed @ 60Hz
Display A, A, B, B, C,
C, ...
Temporal Functions
Display
Eye
Temporal Functions
Display
Eye
0
1/60 s
1/30 s
0
1/60 s
1/30 s
Слайд 14Motion Blur
Helps to Remedy “Flickering”, but…
Happens in the Eye…
Not the camera
(virtual)
Should not be used at all
At eye-trackable speeds
Should not be used at all
At eye-trackable speeds
Observed @ 60Hz
Слайд 19
The Idea is very simple …
Render Velocity Buffer
Interpolate the Middle Frame
Present
at the Right Moment in Time
VBlank
VBlank
VBlank
1/60 s
1/60 s
Слайд 20Ideally (to be “free”)
Reuse as Much as Possible
Velocity buffers
Previous frames @
lower-res
Any other data available (e.g. animations)
No Extra
Latency
Memory / performance hit
Any other data available (e.g. animations)
No Extra
Latency
Memory / performance hit
Слайд 21
Typical Rendering Pipeline
Depth Buffer (~2-3 ms)
Shadows, Lighting (deferred), SSAO
Main Color Pass
& Alpha Pass
Post Effects (HDR, DOF, Motion blur, DLAA, etc...)
Post Effects (HDR, DOF, Motion blur, DLAA, etc...)
Слайд 22
Modified Rendering Pipeline
Depth Buffer (~2-3 ms)
Render Velocity / Interpolate / Schedule
to swap
Do the Rest of the Current Frame Pn
Do the Rest of the Current Frame Pn
GPU is working on
Слайд 23One-frame Based Solution
Pros
No extra latency
Little extra memory
Cons
Issues with shadows and reflections
Manageable
Harder
to fix interpolation errors
But possible ;)
But possible ;)
Слайд 28Very Basic Interpolation III
Based on Current Velocity
Pn-½
= f( Pn-1, -½·Vn )
Sample half way backwards
Previous frame Pn-1
Two Types of Issues
Static geometry tails
Dynamic geometry tails
Sample half way backwards
Previous frame Pn-1
Two Types of Issues
Static geometry tails
Dynamic geometry tails
Слайд 31Artifact Minimization
Static Geometry
Velocity buffer filtering
Dynamic Geometry (Characters)
Render separately, re-render completely or
...
Use Previous Color & Velocity Frames
To fix-up invalid regions
Two-frame solution works best
Use Previous Color & Velocity Frames
To fix-up invalid regions
Two-frame solution works best
Слайд 32Two-frame Based Solution
Pros
Smoother shadows, alpha, reflections
More real data to handle overlaps
Cons
Extra
frame of latency
More memory and slower
More memory and slower
Слайд 33Modified Velocity Rendering
Re-render Characters
At interpolated positions
Use other Channels
To help with interpolation
and artifact detection
B – 8bit depth buffer
A – 8bit id masks (e.g. characters)
B – 8bit depth buffer
A – 8bit id masks (e.g. characters)
Слайд 34Velocity Merging
Very Simple and Efficient
Vmerged = Vn-1 if ( Zn-1 +
∂ ) < Zn
else Vn
Z is embedded in V
Works well for background objects
Assuming continuity of motion
Z is embedded in V
Works well for background objects
Assuming continuity of motion
Слайд 38Character Removal I
Visual System’s Blind Spot Inspired
Synthesize the Interior
Assuming That
Horizontal remains horizontal
Vertical
remains vertical
Texture is preserved
Shading changes gradually
Texture is preserved
Shading changes gradually
Слайд 43Character Removal II
Generate Mask
Leak Neighboring Image Patches
Duplicate and offset Up
Down
Left
Слайд 44Character Removal II
Generate Mask
Leak Neighboring Image Patches
Duplicate and offset Up
Down
Left
Right
Merge all layers
together
Слайд 45Character Removal III
Repeat N Times
Blur the Interior Slightly
original
mask pass 1 pass 2 pass 3 finalize
Слайд 46Character Removal IV
Original
Synthesized
3 passes
@ 640x360
XBox360 ~ 0.4ms
PS3 ~ 0.3ms
(5 SPUs)
3 passes
@ 640x360
XBox360 ~ 0.4ms
PS3 ~ 0.3ms
(5 SPUs)
Слайд 47Improved Interpolation
Based on Vmerged
Sample half way backwards
Previous real frame Pn-1
Synthesized frame
P'n-1
Based on Masks in Vn and P'n-1
Select P'n-1, if it had a character where Vn does not
Else select Pn-1
Based on Masks in Vn and P'n-1
Select P'n-1, if it had a character where Vn does not
Else select Pn-1
Слайд 51
Typical Flipping
Flip Normal Buffer
As soon as VBlank is received
Immediately, if VBlank
passed (frame rate drops)
Слайд 53Custom flipping on XBox360
Bypass Direct3D Present with...
Asynchronous Swaps
Multiple front buffers
SwapCallback, VerticalBlankCallback
Manual
Hardware Manipulation
D1-GRPH_PRIMARY_SURFACE_ADDRESS 0x7FC86110 (undocumented)
D1-GRPH_PRIMARY_SURFACE_ADDRESS 0x7FC86110 (undocumented)
Слайд 54Real-time Implementation
One-frame Based
Interpolation of Dynamic Objects
Character removal
Performance @ 1280x720
XBox360 ~ 1.5
ms
PS3 (5 SPUs) ~ 1.2 ms
PS3 (5 SPUs) ~ 1.2 ms
Слайд 55Motion Blur In TFU2
Current Solution
Alpha “friendly”, simple edge condition
Better than plain
30 fps
But not as good as 60 fps
Performance @ 1280x720
XBox360 ~ 2.2 ± 0.4 ms (5-11 samples)
PS3 (5 SPUs) ~ 1.4 ± 0.5 ms (16 samples)
But not as good as 60 fps
Performance @ 1280x720
XBox360 ~ 2.2 ± 0.4 ms (5-11 samples)
PS3 (5 SPUs) ~ 1.4 ± 0.5 ms (16 samples)
Слайд 56Future Work
Transparency Customizations
Re-render lightsabers, HUD
Multi-layer interpolation
Up-scale from Lower Frame Rates
E.g. render
environment at 20 or 15 fps
Interpolate Shadows and Reflections (or blur)
Predictive Performance Adjustment
Interpolate Shadows and Reflections (or blur)
Predictive Performance Adjustment
