Movatterモバイル変換

Ivano MalavoltaRESEARCHin software engineeringVRIJEUNIVERSITEITAMSTERDAM

HelloEmpirical software engineering+ Software Architecture + MDE+applied toComplex systemsAutonomous robotsMobile-enabled systemsIf you think good architecture is expensive,try bad architecture.... Brian Foote and Joseph Yoder

Course overview (from my perspective)Module name #hours InstructorIntroduction to the course 2 InverardiSE principles, Eclipse IDE, MDE 8 IovinoResearch in software engineering 2 MalavoltaCollaborative software development 2 MalavoltaSoftware architecture 6 MalavoltaModern development paradigms 2 MalavoltaCode generation, applications of MDE, LAB, etc. 10 IovinoPrinciples of software testing 4 BertolinoHomework 1Christmas breakHomework 2

Homework 1Tasks• create an AADL specification describing the architecture of a chosen system• write an Acceleo program that takes as input the AADL model and producesan HTML5 report or other artifact• write a report describing the performed activitiesGoals• to understand what are the advantages of SE principles like abstraction andseparation of concerns• to concretely understand what architectural modeling means• to be exposed to concerns and issues related to dependability• to understand how to use MDE techniques like model transformations andcode generatorsTentative deadline20/12/2016

Homework 2Tasks• pick a set of articles related to a chosen research theme• carefully read them and explore the state of the art about the researchtheme• write a report about your findings• make a presentation to the classroomGoals• to study a specific area of software engineering that may be of interest toyou• to design a scientific paper• to train yourself in reading and writing scientific papersTentative deadline20/01/2017Research themes willbe available soon

RoadmapSoftware engineering researchEmpirical strategiesWriting good research papers

The application of engineering to softwareWe deal with software systems that are:– large and complex– built by teams– exist in many versions– last many years– undergo changesProgramming skills are not enough“Physicist example”Software engineeringProgrammer:complete programSoftware engineer:software component

Software engineering VScomputer scienceComputer Science– Computability, algorithms and complexity, programminglanguages, data structures, databases, artificial intelligence,etc.Software Engineering– The APPLICATION of computer science, mathematics,project management to build high quality software

My vision about researchResearchTheory Industrial projectsProgramming ExperimentationIvano Malavolta. Research Statement. November 2013. http://goo.gl/99N5AS

Research objectiveKey objectives• Quality àutility + functional correctness• Cost à development + use• Timeliness à good-enough result, when it’s neededAddress problems that affect practical softwareReal worldpractical PROBLEMReal worldpractical SOLUTION?

Research strategyReal worldpractical PROBLEMReal worldpractical SOLUTIONResearch settingIDEALIZED PROBLEMResearch settingSOLUTION toIDEALIZED PROBLEMResearch product(technique, method,model, system, …)

Validation of the resultsReal worldpractical PROBLEMReal worldpractical SOLUTIONResearch settingIDEALIZED PROBLEMResearch settingSOLUTION toIDEALIZED PROBLEMResearch product(technique, method,model, system, …)

Validation of the resultsReal worldpractical PROBLEMReal worldpractical SOLUTIONResearch settingIDEALIZED PROBLEMResearch settingSOLUTION toIDEALIZED PROBLEMResearch product(technique, method,model, system, …)Validation task 1Does the productsolve the idealized problem?

Validation of the resultsReal worldpractical PROBLEMReal worldpractical SOLUTIONResearch settingIDEALIZED PROBLEMResearch settingSOLUTION toIDEALIZED PROBLEMResearch product(technique, method,model, system, …)Validation task 1Does the productsolve the idealized problem?Validation task 2Does the producthelp to solve the practical problem?

Validation of the results: example

SE research processResearchquestionsResearchvalidationResearchresults

Types of research questionsFEASIBILITYCHARACTERIZATIONMETHOD/MEANSGENERALIZATIONDISCRIMINATIONDoes X exist, and what is it?Is it possible to do X at all?What are the characteristics of X?What exactly do we mean by X?What are the varieties of X, and how arethey related?How can we do X?What is a better way to do X?How can we automate doing X?Is X always true of Y?Given X, what will Y be?How do I decide whether X or Y?

SE research processResearchquestionsResearchresultsResearchvalidation

Research resultsReal worldpractical PROBLEMReal worldpractical SOLUTIONResearch settingIDEALIZED PROBLEMResearch product(technique, method,model, system, …)

Types of research resultsQUALITATIVE &DESCRIPTIVEMODELSTECHNIQUESSYSTEMEMPIRICALMODELSANALYTICMODELSReport interesting observationsGeneralize from (real-life) examplesStructure a problem area; ask good questionsInvent new ways to do some tasks, includingimplementation techniquesDevelop ways to select from alternativesEmbody result in a system, using the systemboth for insight and as carrier of resultsDevelop empirical predictive models fromobserved dataDevelop structural models that permit formalanalysis

Research validationReal worldpractical PROBLEMReal worldpractical SOLUTIONResearch settingIDEALIZED PROBLEMResearch settingSOLUTION toIDEALIZED PROBLEMResearch product(technique, method,model, system, …)Validation task 1Does the productsolve the idealized problem?Validation task 2Does the resulthelp to solve the practical problem?

Types of research validationPERSUASION I thought hard about this, and I believe…IMPLEMENTATION Here is a prototype of a system that…EVALUATION Given these criteria, the object rates as…ANALYSIS Given the facts, here are consequences…Formal model Rigorous derivation and proofEmpirical model Data on use in controlled situationEXPERIENCE Report on use in practiceQualitative model NarrativeDecision criteria Comparison of systems in actual useEmpirical model Data, usually statistical, on practice

Building blocks for researchFeasibilityCharacterizationMethod/meansGeneralizationSelectionQualitative modelTechniqueSystemEmpirical modelAnalytic modelPersuasionImplementationEvaluationAnalysisExperienceQuestion Result Validation

Is this a good plan?FeasibilityCharacterizationMethod/meansGeneralizationSelectionQualitative modelTechniqueSystemEmpirical modelAnalytic modelPersuasionImplementationEvaluationAnalysisExperienceQuestion Result Validation

A common good planFeasibilityCharacterizationCan X bedone better?GeneralizationSelectionQualitative modelTechniqueBuild YEmpirical modelAnalytic modelPersuasionImplementationMeasure Y,compare to XAnalysisExperienceQuestion Result Validation

A common, but bad, planFeasibilityCharacterizationMethod/meansGeneralizationSelectionQualitative modelTechniqueSystemEmpirical modelAnalytic modelPersuasionImplementationEvaluationAnalysisExperienceQuestion Result Validation

Two other good plansCan X be doneat all?CharacterizationIs X alwaystrue of Y?SelectionQualitative modelTechniqueBuild a Ythat does XEmpirical modelFormally modelY, prove X“Look it works!”ImplementationCheck proofExperienceQuestion Result ValidationMethod/means Evaluation

LABChoose a research paper and try to map it into the buildingblocks of SE researchFeasibilityCharacterizationMethod/meansGeneralizationSelectionQualitative modelTechniqueSystemEmpirical modelAnalytic modelPersuasionImplementationEvaluationAnalysisExperienceQuestion Result Validation

“NO-NO”s for software engineeringresearch• Assume that a result demonstrated fro a 10K-line systemwill scale to a 500K-line system• Expect everyone to do things “my way”• Believe functional correctness is sufficient• Assume the existence of a complete, consistentspecification• Just build things without extracting enduring lessons• Devise a solution in ignorance of how the world reallyworks

How do you trust a research then?1. What are the problems from the real world?– Are they “real” and widespread?2. Are the solutions general? What are their limits?Real worldpractical PROBLEMReal worldpractical SOLUTION?EMPIRICAL SOFTWARE ENGINEERING

Empirical strategies**We will have an advanced course on this topic

Empirical software engineeringScientific use of quantitative and qualitative data to– understand and– improvesoftware products and software development processesData is central to address any research questionIssues related to validity addressed continuously[Victor Basili]

Intuition• You apply the scientific method– ask yourself a question– observe a phenomenon– formulate an hypothesis– build a prediction– setup an experiment– perform analysis on your results

ExampleEnd Users’ Perception of Hybrid Mobile Appsin the Google Play StoreIvano Malavolta, Stefano RubertoTommaso Soru, Valerio TerragniIvano MalavoltaGran Sasso Science Institute (Italy)ivano.malavolta@gssi.infn.itABSTRACTRecently, companies like IBM and Adobe and a growingcommunity of developers advocate hybrid mobile appsdevelopment as a possible solution to mobile platformsfragmentation. Hybrid mobile apps are consistent acrossplatforms and built on web standards.In this study, we present an empirical investigation into mobilehybrid apps. Our goal is to identify and analyse the traits anddistinctions of publicly available hybrid mobile apps from endusers’ perspective. The study has been conducted by mining11,917 free apps and 3,041,315 reviews from the Google PlayStore, and analyzing them from the end users’ perceptionperspective. The results of this study build an objective andreproducible snapshot about how hybrid mobile development isperforming “in the wild” in real projects, thus establishing a basefor future methods and techniques for developing hybrid mobileapps.FINDINGS•  hybrid development frameworks are perceived as bettersuited for data-intensive mobile apps, whereas theyperform poorly when dealing with low-level, platform-speciﬁc features•  end users value hybrid and native apps similarly•  in some categories, end users perceive native apps betterthan hybrid apps with respect to performance and thepresence of bugsRESEARCH QUESTIONSWhat is the difference between hybrid and native mobile appsas perceived by end users?– RQ1: What is the difference in the perceived value betweenhybrid and native mobile apps?– RQ2: What is the difference in the perceived performancebetween hybrid and native mobile apps?– RQ3: What is the difference in the perceived bugginessbetween hybrid and native mobile apps?– RQ4: What is the difference in the initial download overheadbetween hybrid and native mobile apps?

Dimensions of empirical studies“In the lab” versus “in the wild” studiesQualitative versus quantitative studiesPrimary versus secondary studies

“In the lab” versus “in the wild”studiesCommon “in the lab” methods– Controlled experiments– Literature reviews– SimulationsCommon “in the wild” methods– Quasi-experiments– Case studies– Survey research– Ethnographies– Action research

Qualitative versus quantitativestudiesQualitative researchstudying objects in their natural setting and letting thefindings emerge from the observations– inductive process– the subject is the personQuantitative researchquantifying a relationship with the aim to identify a cause-effect relationship– fixed implied factors– focus on collected quantitative data à promotes comparison andstatistical analysesThey arecomplementary

Primary versus secondary studiesPrimary studiesempirical studies in which we directly make measurementsor observations about the objects of interest, whether bysurveys, experiments, case studies, etc.Secondary studiesempirical studies that do not generate any data from directmeasurements, but:– analyze a set of primary studies– usually seek to aggregate the results from these in order toprovide stronger forms of evidence about a phenomenon

Types of empirical studies• Survey• Case study• Experiment

SurveyDef: a system for collecting information from or aboutpeople to describe, compare or explain their knowledge,attitudes and behaviorOften an investigation performed in retrospectInterviews and questionnaires are the primary means ofgathering qualitative or quantitative dataThese are done through taking a sample which isrepresentative from the population to be studied

Example: our survey on arch. languages1. ALs Identification– Definition of a preliminary set of ALs– Systematic search2. Planning the Survey3. Designing the survey4. Analyzing the Data– vertical analysis (and coding) + horizontal analysis

Case studyDef: an empirical enquiry to investigate one instance (or asmall number of instances) of a contemporary softwareengineering phenomenon within its real-life context,especially when the boundary between phenomenon andcontext cannot be clearly specifiedObservational studyData collected to track a specific attribute or establishingrelationships between different attributesMultivariate statistical analysis is often applied

ExperimentDef: an empirical enquiry that manipulates one factor orvariable of the studied setting1. Identify and understand the variables that play a role in softwaredevelopment, and the connections between variables2. Learn cause-effect relationships between the developmentprocess and the obtained products3. Establish laws and best practices about software construction

Examplehttp://dl.acm.org/citation.cfm?id=2491411.2491428

Examples of empirical studies in SE• Nejati, Javad, and Aruna Balasubramanian. "An in-depth study ofmobile browser performance." Proceedings of the 25th InternationalConference on World Wide Web. International World Wide WebConferences Steering Committee, 2016.• Hassan, Safwat, Weiyi Shang, and Ahmed E. Hassan. "An empiricalstudy of emergency updates for top android mobile apps." EmpiricalSoftware Engineering (2016): 1-42.• Tufano, Michele, et al. "An empirical investigation into the nature oftest smells." Proceedings of the 31st IEEE/ACM InternationalConference on Automated Software Engineering. ACM, 2016.• Linares-Vásquez, Mario, et al. "Mining energy-greedy api usagepatterns in android apps: an empirical study." Proceedings of the 11thWorking Conference on Mining Software Repositories. ACM, 2014.• Procaccianti, Giuseppe, Héctor Fernández, and Patricia Lago."Empirical evaluation of two best practices for energy-efficientsoftware development." Journal of Systems and Software 117 (2016):185-198.

How to have an impact in reality?This is called technology transfer

Writing good softwareengineering papers

Let’s reconsider our SE researchprocess…ResearchquestionsResearchresultsResearchvalidation

Research questionsThe program committee looks for– a clear statement of the specific problem you solved– the question about software development you answered– an explanation of how the answer will help solve an importantsoftware engineering problemYou'll devote most of your paper to describing your result,but you should begin by explaining what question youare answering and why the answer mattersResearchquestions

Research resultsExplain precisely– what you have contributed to the catalogue of softwareengineering knowledge– how this is useful beyond your own project

Why should the reader believe yourresult?Show evidence that your result• is valid• helps to solve the problem

A couple of words on the abstract ofa paper• People judge papers by their abstracts– usually decide whether to read the whole paper based onabstract• It's important for the abstract to tell the whole story– background and context– goals– method– results– conclusion• Often it is the only part freely accessible

Example of an abstract structure:1. Two or three sentences about the current state of the art,identifying a particular problem2. One or two sentences about what this paper contributes toimproving the situation3. One or two sentences about the specific result of the paperand the main idea behind it4. A sentence about how the result is demonstrated ordefended

Coming back to the initial example…State ofthe artOverallcontributionSpecificresultsValidation✗ ✓✓✗ ✗

Second try…State ofthe artOverallcontributionSpecificresultsValidation

What this lecture means to you?Software engineering != computer scienceYou now know how to carry on research in SEDon’t focus on the “size” of the problem, but on– the relevance (the practical, but also the theoretical!)– the accuracy in the investigation (problem and evaluationresearch)When conducting empirical research, don’t make claimsyou cannot eventually measure

Suggested readings about scientificwriting• Shaw, M. (2003), Writing Good Software Engineering ResearchPapers, in Lori A. Clarke; Laurie Dillon & Walter F. Tichy, ed., 'ICSE' ,IEEE Computer Society, , pp. 726-737 .• Shaw, M. (2002), 'What makes good research in softwareengineering?', STTT 4 (1) , 1-7 .

Referenceshttp://link.springer.com/book/10.1007%2F978-3-642-29044-2

ContactIvano Malavolta |Assistant professorVrije Universiteit Amsterdamiivanooi.malavolta@vu.nlwww.ivanomalavolta.com

AcknowledgementSome parts of this lecture are inspired by:• Ivica Crnkovic’s lecture on software engineering research atMälardalen University (Sweden)• Matthias Galster ‘s tutorial titled “Introduction to Empirical ResearchMethodologies” at ECSA 2014

Movatterモバイル変換

Change Language

[2016/2017] RESEARCH in software engineering

Embed presentation

Recommended

More Related Content

What's hot

Viewers also liked

Similar to [2016/2017] RESEARCH in software engineering

More from Ivano Malavolta

Recently uploaded

[2016/2017] RESEARCH in software engineering