NotificationsYou must be signed in to change notification settings
Fork45
Star142

New fuzzing platform#1457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Merged

Markoutte merged 9 commits intomainfrompelevin/fuzzing_platform

Dec 12, 2022

Merged

New fuzzing platform#1457

Markoutte merged 9 commits intomainfrompelevin/fuzzing_platform

Dec 12, 2022

Conversation

Copy link

Collaborator

Markoutte commentedDec 6, 2022•
edited
Loading

Description

Adds new fuzzing platform that simplifies developing fuzzing for new languages. Old implementation is still in codebase but will be deprecated and removed soon. Only base Java implementation migrated to the new fuzzing platform at the moment.

Please, seeFuzzing Platform Design doc for understanding of basic concepts. In nutshell now any work with fuzzing requires some 'seeds' or 'tasks' which can be 4 types:

Simple seed is just a value with optional mutator
Known [structure] seed is something that has common structure for different types, e.g. integers (byte, short, int, long) which can be represented as a bit vector. This type can be used instead of 1 to faster developing of defaults such as integers, floats, strings, etc.
Recursive seed can represent some recursive structures that require to fuzz some parameters before constructing itself.
Collection seed represents collections such as arrays, lists, sets, maps and others.

There aresome demos that clarify these concepts.

This platform is fully implemented for Java and plugin can be tested withthese samples.

Fixes#1448

Also note, that after short discussion this implementation doesn't use mocks because it looks wrong concepts in terms of fuzzing. Fuzzing doesn't analyze the code therefore it cannot supply correct mock calls for tests. Therefore#747 is discontinued starting this PR.

Type of Change

Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Regression and integration tests

Test samples for manual testing were added. Integration test for fuzzing is being under developing.

Automated Testing

All tests from theutbot-fuzzer andutbot-fuzzing modules must pass.

Manual Scenario

Samples for manual scenario.

Checklist:

The change followed the style guidelines of the UTBot project
Self-review of the code is passed
The change contains enough commentaries, particularly in hard-to-understand areas
New documentation is provided or existed one is altered
No new warnings
New tests have been added
All tests pass locally with my changes

Markoutte requested review fromSBOne-Kenobi,Vassiliy-Kudryashov,amandelpie,denis-fokin,tamarinvs19 andvolivan239

December 6, 2022 05:47

denis-fokin reviewed

Dec 6, 2022

View reviewed changes

docs/Fuzzing Platform.md OutdatedShow resolvedHide resolved

Markoutte requested a review fromdenis-fokin

December 7, 2022 08:13

amandelpie approved these changes

Dec 7, 2022

View reviewed changes

denis-fokin reviewed

Dec 7, 2022

View reviewed changes

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt OutdatedShow resolvedHide resolved

denis-fokin reviewed

Dec 7, 2022

View reviewed changes

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt Outdated

		)
		}

		privatefunshouldPass(type:ClassId):Boolean {

Copy link

Collaborator

denis-fokinDec 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

What "should pass" means in this context?

Copy link

CollaboratorAuthor

MarkoutteDec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Changed toisIgnored(type: ClassId)

denis-fokin reviewed

Dec 7, 2022

View reviewed changes

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt Outdated

		valgetter:Method?
		)

		internalfunfindSuitableFields(classId:ClassId,packageName:String?):List<FieldDescription> {

Copy link

Collaborator

denis-fokinDec 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I would add a piece of documentation here. Suitable for what?

Copy link

CollaboratorAuthor

MarkoutteDec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Changed tofindAccessibleModifableFields(...)

Markoutte force-pushed thepelevin/fuzzing_platform branch from6e39690 to97d8336Compare

December 8, 2022 05:12

Markoutte requested a review fromdenis-fokin

December 8, 2022 05:20

denis-fokin reviewed

Dec 8, 2022

View reviewed changes

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/Api.kt Outdated

		* @param description contains user-defined information about current run. Can be used as a state of the run.
		* @param values current values to run.
		*/
		suspendfunrun(description:DESCRIPTION,values:List<RESULT>):FEEDBACK

Copy link

Collaborator

denis-fokinDec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Please, find a better naming for the method. 'Run' makes me think that the method is supposed to be called only ones. I would suggest something like: 'hanlde', 'valueUdated' or something similar

Markoutte requested a review fromdenis-fokin

December 8, 2022 12:00

SBOne-Kenobi approved these changes

Dec 8, 2022

View reviewed changes

Copy link

Collaborator

SBOne-Kenobi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Amazing! LGTM :)

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/Api.kt

		builder=PassRoutine("Main Routine"),
		state=State(1, typeCache),
		)
		val dynamicallyGenerated= mutableListOf<Node<T,R>>()

Copy link

Collaborator

SBOne-KenobiDec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

As I understand you use it as a queue, can you replace it with any instance of queue such as LinkedList or Deque.

Markoutte added8 commits

December 12, 2022 07:19

New fuzzing platform

a809321

Fixes for summarization

fddb530

Fix fuzzing when collection with generics is this object

5423727

Fix for summarization when list of parameters was the same for every …

6f9fa08

…case

Changes to fuzzer doc

cc89dc5

Clarify some ObjectValueProvider entities

3897c5e

Fuzzer can pass one object into several arguments of same type

47a4c3c

Fix double values mutations

aa239b8

volivan239 approved these changes

Dec 12, 2022

View reviewed changes

Copy link

Collaborator

volivan239 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Looks monumental!

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/demo/AbcFuzzing.kt

Comment on lines +13 to +19

		fun String.findMaxSubstring(s:String) :Int {
		if (s.isEmpty())return-1
		for (iin s.indices) {
		if (s[i]!=this[i])return i-1
		}
		return s.length
		}

Copy link

Collaborator

volivan239Dec 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think the name is misleading, because the function looks like a strange variation ofCharSequence.commonPrefixWith()

Also, it may throwStringIndexOutOfBoundsException ifthis is a strict prefix ofs. If this is intentional, please, add a comment describing how fuzzing handle exceptions in this example.

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/Configuration.kt

		/**
		* Probability of creating shifted array values instead of generating new values for modification.
		*/
		varprobCollectionMutationInsteadCreateNew:Int =50,

Copy link

Collaborator

volivan239Dec 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Here and below: imo, if probability is not in[0, 1] it is worth mentioning that it is in percents

utbot-fuzzing/src/main/kotlin/org/utbot/fuzzing/utils/Functions.kt Outdated

Comment on lines 88 to 95

		funmain() {
		val endian=Endian.BE
		println(255.toUByte().toBinaryString(endian))
		println(2.toBinaryString(endian))
		println(BitVectorValue.fromInt(2).toBinaryString(endian))
		print(8.75f.toBinaryString(endian))
		print(8.75.toBinaryString(endian))
		} No newline at end of file

Copy link

Collaborator

volivan239Dec 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think this should be either moved to tests or deleted. Same for IEEE754Value.kt

utbot-framework/src/main/kotlin/org/utbot/engine/UtBotSymbolicEngine.kt

		TrieBasedFuzzerStatistics(coveredInstructionValues), methodUnderTestDescription,*defaultModelMutators().toTypedArray()
		)
		fuzzedValues.forEach { values->
		val names= graph.body.method.tags.filterIsInstance<ParamNamesTag>().firstOrNull()?.names?: emptyList()

Copy link

Collaborator

volivan239Dec 12, 2022•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thisnames initialization looks weird. Can't we just usemethodUnderTest.parameters.map { it.name } here? If not, please, add comment reasoning such an initialization