W E B I N A R
- Главная
- Разное
- Дизайн
- Бизнес и предпринимательство
- Аналитика
- Образование
- Развлечения
- Красота и здоровье
- Финансы
- Государство
- Путешествия
- Спорт
- Недвижимость
- Армия
- Графика
- Культурология
- Еда и кулинария
- Лингвистика
- Английский язык
- Астрономия
- Алгебра
- Биология
- География
- Детские презентации
- Информатика
- История
- Литература
- Маркетинг
- Математика
- Медицина
- Менеджмент
- Музыка
- МХК
- Немецкий язык
- ОБЖ
- Обществознание
- Окружающий мир
- Педагогика
- Русский язык
- Технология
- Физика
- Философия
- Химия
- Шаблоны, картинки для презентаций
- Экология
- Экономика
- Юриспруденция
Learn the Best Practices in Mobile Game Testing презентация
Содержание
- 1. Learn the Best Practices in Mobile Game Testing
- 2. Agenda Different Types of Testing and How
- 3. Let's Define 'Testing'...
- 4. Different Types of Testing? Black-box testing vs.
- 5. Defining Some Ways to Test Feature-based testing Performance testing End-user testing
- 6. The Old-Fashion Thinking Deterministic systems Identifiable
- 7. Test Automation as a Solution Random setups/results -> Easy
- 8. Challenges that Test Automation Tackles Proliferation of
- 9. OS versions Chipsets CPU + GPU Tens
- 10. Correct behaviour across platforms and browsers
- 11. Importance of Mobile Game Testing
- 12. Important Aspects in Game Testing User Interface
- 13. User Interface and Functional Compared to
- 14. Performance Testing Performance, Load, Stress, Spike, Soak
- 15. Studies: Majority of users leave game due
- 16. Building the User Experience
- 17. End-to-end between tests, AI or more simple
- 18. Important part of the after-engagement (how gamers
- 19. Security and Liabilities
- 20. 4 Top Priorities in Mobile Game Testing
- 21. Localization
- 22. Device Diversity By OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/
- 23. Real Devices vs. Emulators Emulators/simulators won’t help
- 24. Manual vs. Automation Smaller coverage, More money
- 25. Infrastructure for Mobile Game Testing
- 26. High-Level Infrastructure
- 27. Architecture / Infrastructure GAME Game Engine Android
- 28. Isolated, Hermetic Testing GAME Game Engine Android
- 29. Compatibility & Porting Testing GAME Game Engine
- 30. Smoke/Reference Testing GAME Game Engine Android iOS
- 31. Back-End Performance Testing GAME Game Engine Android
- 32. Continuous Integration
- 33. Game Testing Process
- 34. Hands-on Example: Integrating Development with Testing Environment
- 35. Continuous Integration (e.g. Jenkins) Appium framework (for
- 36. Continuous Integration
- 37. Appium Intro Family Tree: Android Test Automation Frameworks Appium works for both, Android and iOS
- 38. © Copyrights by Bitbar Technologies Ltd. 2013
- 39. © Copyrights by Bitbar Technologies Ltd. 2014
- 40. © Copyrights by Bitbar Technologies Ltd. 2014
- 41. © Copyrights by Bitbar Technologies Ltd. 2014
- 42. © Copyrights by Bitbar Technologies Ltd. 2014
- 43. © Copyrights by Bitbar Technologies Ltd. 2014
- 44. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Example
- 45. 350+ Android & 20+ iOS Devices
- 46. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Example
- 47. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Example
- 48. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Example
- 49. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Example
- 50. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Example
- 51. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Example
- 52. Example Projects and Test Runs
- 53. Appium Broker © Copyrights by Bitbar Technologies
- 54. Image Recognition The basic example of Image Recognition in Mobile Game Testing: Template Matching
- 55. Code Example
- 56. Code Example
- 57. Code Example
- 58. Code Example
- 59. Test Script contains the following 'steps': Identify
- 60. VIDEO Image Recognition with Real Android and iOS Devices in Local Environment
- 61. Testdroid Update
- 62. New UI in Testdroid Cloud 2.0
- 63. Testdroid Cloud's Interactive
- 64. Testdroid PrivateCloud - Running user-selected devices at any of our hosting facilities
- 65. NEW HOSTING CENTER! Mountain View, CA
- 66. Option 1: Public Device Cloud on-demand
- 68. Video recording & slides of this
- 69. We operate the largest global device cloud
Agenda Different Types of Testing and How Game Testing Differs from Regular App Testing? Frameworks, Tools and Technologies – What is Important in Mobile Game Testing Today? Top Priorities for
Слайд 1
2 December 2014
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com
Learn the Best Practices in
Mobile Game Testing
Слайд 2Agenda
Different Types of Testing and How Game Testing Differs from Regular
App Testing?
Frameworks, Tools and Technologies – What is Important in Mobile Game Testing Today?
Top Priorities for Mobile Game Testing
Infrastructure for Successful Game Testing
Hands-on Example (Image Recognition)
Testdroid Update
Q&A
Frameworks, Tools and Technologies – What is Important in Mobile Game Testing Today?
Top Priorities for Mobile Game Testing
Infrastructure for Successful Game Testing
Hands-on Example (Image Recognition)
Testdroid Update
Q&A
Слайд 4Different Types of Testing?
Black-box testing vs. White-box testing
Methods used in Mobile
Game Testing
Functional
Compatibility
Performance
Localization
Regression
Load
Smoke
Acceptance
Functional
Compatibility
Performance
Localization
Regression
Load
Smoke
Acceptance
Слайд 6The Old-Fashion Thinking
Deterministic systems
Identifiable objects
Tool support(s)
User-driven actions
Simple inputs/outputs
Virtualization
Random setups/results
No handle on
objects
No standards
User reacts to input
Complex inputs/outputs
Real hardware
No standards
User reacts to input
Complex inputs/outputs
Real hardware
<>
<>
<>
<>
<>
<>
High-level of automation
Low-level of automation
X
Слайд 7Test Automation as a Solution
Random setups/results -> Easy to generate random events
No handle
on objects -> Variety of options (e.g. image recognition)
No standards -> Variety of frameworks (cross-platform etc.)
User reacts to input -> Test scripts can handle random events
Complex inputs/outputs -> Test scripts can generate input/content
Real hardware -> Availability of real mobile hardware
No standards -> Variety of frameworks (cross-platform etc.)
User reacts to input -> Test scripts can handle random events
Complex inputs/outputs -> Test scripts can generate input/content
Real hardware -> Availability of real mobile hardware
Test Automation brings real value
with variety of options to mobile game testing
Слайд 8Challenges that Test Automation Tackles
Proliferation of Devices, Brands, Operators, Platforms
Coupling the
Testing and Development process (Continuous Integration/Deployment/Delivery)
Different Types of Testing (as automated)
Stress, Load, Performance, Functionality etc.
Test script & test asset usability
Cons of Manual testing
Tedious effort, Expensive human resources, Error-prone (as done by humans) & Not reusable (done once)
Different Types of Testing (as automated)
Stress, Load, Performance, Functionality etc.
Test script & test asset usability
Cons of Manual testing
Tedious effort, Expensive human resources, Error-prone (as done by humans) & Not reusable (done once)
Слайд 9OS versions
Chipsets
CPU + GPU
Tens of OEMs
Memory
Displays (resolutions, physical hw)
OEM mods
Other
hardware
(connectivity, calibration)
(connectivity, calibration)
Relation to other software
Significant Benefits of Using Automation
Слайд 10Correct behaviour across platforms and browsers
Integration with web back-ends
Typically need to
fully utilize HW (CPU+GPU)
Resource (e.g. battery) consumption
OpenGL ES 2/3
Resource (e.g. battery) consumption
OpenGL ES 2/3
Functionality and usability
Screen orientations, connectivity, user profiles
Robustness
Robustness and security!
Brand
Compliances, verification with back-ends and data
Different 'Verticals' Require Different Practices
Слайд 12Important Aspects in Game Testing
User Interface and Functionality
UI layouts/elements, Menus, Resolution,
Orientation, Overall visuality
Graphics Performance
Usability and User Experience
Responsiveness, Real Occuring Events
Multi-player/User
Social Integrations
Security and Liabilities
Graphics Performance
Usability and User Experience
Responsiveness, Real Occuring Events
Multi-player/User
Social Integrations
Security and Liabilities
Слайд 13User Interface and Functional
Compared to regular app testing, mobile game
testing may seem to be more challenging
e.g. OpenGL ES content not action-identifiable
Graphics and UI – Identical look&feel across all devices and variants
Usability controls in the context of game
Navigation/game-play, Progress, Delays between "game-stages"
Animations, Fonts, Other graphics assets
Screen Resolution and Orientation
e.g. OpenGL ES content not action-identifiable
Graphics and UI – Identical look&feel across all devices and variants
Usability controls in the context of game
Navigation/game-play, Progress, Delays between "game-stages"
Animations, Fonts, Other graphics assets
Screen Resolution and Orientation
Слайд 14Performance Testing
Performance, Load, Stress, Spike, Soak
Measure the real user experience
Execute tests
with all possible devices (SW&HW)
Discover the impact of server loads, app performance (CPU/GPU utilization)
Discover the impact of server loads, app performance (CPU/GPU utilization)
Слайд 15Studies: Majority of users leave game due sluggish, general slowness or
other bad performance behavior
It's very important to understand how game behaves on different devices, under different circumstances
It's very important to understand how game behaves on different devices, under different circumstances
Performance Makes +/- UX
Слайд 16
Building the User Experience
Understanding
How quickly gamer assimilates what the game is
about, how to fully play with it, what are the functionalities.
Usability
The main User Interface, menus, design and how usable those are
Playing Experience
Intuition to follow the game and go forward with its instructions and game-play
Monetization
Are gamers willing to buy – invest – in game
Graphics performance capability is the most important thing for mobile games when forming UX
Too many games fail to work properly with low-end/mid-range devices due hardware performance
The real understanding of game's performance capability can be only figured out by testing on real devices
Слайд 17End-to-end between tests, AI or more simple implementations
Optimize the data communication
between server and clients
Test simultaneously on several (different) devices
Check what happens if game gets interrupted (platform)
Location-based features – how to properly mock up those?
Hermetic environment – what happens?
Test simultaneously on several (different) devices
Check what happens if game gets interrupted (platform)
Location-based features – how to properly mock up those?
Hermetic environment – what happens?
Multi-Player/User Features
Слайд 18Important part of the after-engagement (how gamers share their results)
Many of
OEM branded devices provide certain SoMe apps pre-installed (version?)
Connectivity, login and data transferring are tested in context of game
Hermetic environment – what happens?
Connectivity, login and data transferring are tested in context of game
Hermetic environment – what happens?
Social Integrations
Слайд 22Device Diversity
By OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/
Слайд 23Real Devices vs. Emulators
Emulators/simulators won’t help you to test…
User Experience and
Usability
Hardware
Software
Infrastructure
Hardware
Software
Infrastructure
0 %
= the percentage of your app users that use emulator to run your app!
Why Real Devices are Must-to-Have?
Слайд 24Manual vs. Automation
Smaller coverage, More money burnt & time wasted, Error-prone
Large
coverage, quickly completed, Less money & time wasted, Exact results.
Manual
Automation
Слайд 27Architecture / Infrastructure
GAME
Game Engine
Android
iOS
...
API
&
services
Server
Database
App
Middleware
Platform
Connectivity/Network
Back-end
Слайд 28Isolated, Hermetic Testing
GAME
Game Engine
Android
iOS
...
API
&
services
Server
Database
Hermetic Environment
App
Middleware
Platform
Connectivity/Network
Back-end
Слайд 29Compatibility & Porting Testing
GAME
Game Engine
Android
iOS
...
API
&
services
Server
Database
Real Devices
App
Middleware
Platform
Connectivity/Network
Back-end
Слайд 30Smoke/Reference Testing
GAME
Game Engine
Android
iOS
...
API
&
services
Server
Database
Reference Environment
App
Middleware
Platform
Connectivity/Network
Back-end
Слайд 31Back-End Performance Testing
GAME
Game Engine
Android
iOS
...
API
&
services
Server
Database
Massive no. of users and significant load
App
Middleware
Platform
Connectivity/Network
Back-end
Слайд 35Continuous Integration (e.g. Jenkins)
Appium framework (for Android and iOS)
No need for
devices – we'll use devices on Testdroid Cloud service
Test scripts / test cases for game
Test scripts / test cases for game
Setup
Слайд 37Appium Intro
Family Tree: Android Test Automation Frameworks
Appium works for both, Android
and iOS
Слайд 38© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.
Uses Selenium
Webdriver (W3C standard) as a scripting framework
Supports native Android, native iOS and mobile web:
Android via uiautomator (API level >=16) and Selendroid (API level <16)
iOS via UI Automation
Mobile web as Selenium driver for Android and iOS
You can write your Appium scripts on almost any programming language (Haskell/Go/Clojure/Java/Ruby)
Supports native Android, native iOS and mobile web:
Android via uiautomator (API level >=16) and Selendroid (API level <16)
iOS via UI Automation
Mobile web as Selenium driver for Android and iOS
You can write your Appium scripts on almost any programming language (Haskell/Go/Clojure/Java/Ruby)
Appium Basics
Слайд 39© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
Test Script
Test
Case
Desired Capabilities
Desired Capabilities / Android
Example
Слайд 40© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
Test Script
Test
Case
Desired Capabilities
Example
Desired Capabilities / iOS
Слайд 41© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
Test Script
Test
Case
Desired Capabilities
Appium @ Localhost
WebDriver
http://localhost_4723/wd/hub
Appium Server
4723
Device
Example
Слайд 42© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
Test Script
Test
Case
Desired Capabilities
From Localhost to Testdroid Cloud
WebDriver
http://localhost_4723/wd/hub
*Testdroid Caps
http://appium.testdroid.com/wd/hub
Example
Слайд 43© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
Test Script
Test
Case
Desired Capabilities
From Localhost to Testdroid Cloud
WebDriver
{
“testdroid_username”: “user@domain.com”,
“testdroid_password”: “p4s$w0rd”,
“testdroid_project”: “My First Project”,
“testdroid_testrun”: “Test 1”,
“testdroid_device”: “iPad Mini 7.0.4 A1432”,
“testdroid_app”: “http://domain.com/app_v1.ipa”
.
.
“app”: “com.bitbar.testdroid.BitbarIOSSample”
}
Example
Слайд 53Appium Broker
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
Test
Script
Behind the Scene
Device Cluster
Device 1
Device 2
Device 3
Session Map
Proxy
Appium
Device 1
Example
Слайд 54Image Recognition
The basic example of Image Recognition in Mobile Game Testing:
Template Matching
Слайд 59Test Script contains the following 'steps':
Identify the used platform (Android /
iOS)
Search for 'Goldmine' graphical element (using find_image to search for screen)
Script runs test to go to shop
Script runs test to buy a cannon
Script runs test to place the cannon (all these three are defined in .png files)
Script runs test to start a battle!
Game is brought down and test ends.
Search for 'Goldmine' graphical element (using find_image to search for screen)
Script runs test to go to shop
Script runs test to buy a cannon
Script runs test to place the cannon (all these three are defined in .png files)
Script runs test to start a battle!
Game is brought down and test ends.
Example: Clash of Clans
Слайд 66Option 1:
Public Device Cloud
on-demand devices (multitenant)
Mobile app testing on over 400+
real Android and iOS devices hosted by Bitbar
Option 2:
Private Device Cloud
reserved devices
Hosted by Bitbar in the US and/or Europe
Devices chosen by and reserved only for the
Customer
Option 3:
On-premise Device Cloud
Automated mobile app testing on real Android and iOS de- vices hosted by the customer, usually
30-500 devices
Testdroid Deployment Options
Слайд 68
Video recording & slides of this webinar will be soon available
at Vimeo and Slideshare!
Please visit at www.testdroid.com for more information.
Please visit at www.testdroid.com for more information.
THANK YOU!
W E B I N A R
Слайд 69We operate the largest global device cloud with over 400 devices
instantly
available for developers
Devices from the US, Europe, China, Japan and Korea
– global devices for global customers
Devices from the US, Europe, China, Japan and Korea
– global devices for global customers
