US20240022585A1 - Detecting and responding to malicious acts directed towards machine learning model - Google Patents

Abstract

A system detects and responds to malicious acts directed towards machine learning models. Data fed into and output by a machine learning model is collected by a sensor. The data fed into the model includes vectorization data, which is generated from raw data provided from a requester, such as for example a stream of timeseries data. The output data may include a prediction or other output generated by the machine learning model in response to receiving the vectorization data. The vectorization data and machine learning model output data are processed to determine whether the machine learning model is being subject to a malicious act (e.g., attack). The output of the processing may indicate an attack score. A response for handling the request by a requester may be selected based on the output that includes the attack score, and the response may be applied to the requestor.

Description

BACKGROUND
• Machine learning computing resources are becoming more popular in products and computing systems. With the increased presence of machine learning model resources, so have the attacks perpetrated on machine learning-based systems by bad actors. Traditional methods of virus detection do not detect attacks made against most machine learning systems. As such, what is needed is an improved method for detecting attacks on machine learning systems.
SUMMARY
• The present technology, roughly described, detects and responds to malicious acts directed towards machine learning models. Data fed into and output by a machine learning model is collected by a sensor. The data fed into the model includes vectorization data, which is generated from raw data provided from a requester, such as for example a stream of timeseries data. The output data may include a prediction or other output generated by the machine learning model in response to receiving the vectorization data.
• The vectorization data and machine learning model output data are processed to determine whether the machine learning model is being subject to a malicious act (e.g., attack). The output of the processing may indicate an attack score, for example in the form of a prediction whether the machine learning model is subject to malicious act. An alert may be generated based on the value of the attack score. A response for handling the request by a requester may be selected based on the output that includes the attack score, and the response may be applied to the requestor.
• In some instances, the present technology provides a method for monitoring a machine learning-based system for malicious acts. The method begins by receiving vectorization data to the sensor. The vectorization data is derived from input data intended for a machine learning model and provided by a requestor. The sensor then receives an output generated by the machine learning model, wherein the machine learning model generates the output in response to receiving the vectorization data. Vectorization data and the model output are then processed by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data and model output. A response is applied to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.
• In some instances, a non-transitory computer readable storage medium includes embodied thereon a program, the program being executable by a processor to perform a method for monitoring a machine learning-based system for malicious acts. The method begins with receiving vectorization to the sensor. The vectorization data is derived from input data intended for a first machine learning model and provided by a requestor. The sensor then receives an output generated by the machine learning model, wherein the machine learning model generates the output in response to receiving the vectorization data. Vectorization data and model output are then processed by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data and the model output. A response is applied to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.
• In some instances, a system for monitoring a machine learning-based system for malicious acts includes a server having a memory and a processor. One or more modules can be stored in the memory and executed by the processor to receive vectorization data by a sensor, the vectorization data derived from input data intended for a first machine learning model and provided by a requestor, receive, by the processing engine, an output generated by the machine learning model, the machine learning model generating the output in response to receiving the vectorization data, process the vectorization data and the model output by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data, and apply a response to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.
BRIEF DESCRIPTION OF FIGURES
• FIG.1 is a block diagram of a system for detecting and responding to malicious acts directed towards a machine learning model.
• FIG.2 is a block diagram of a customer data store.
• FIG.3 is a block diagram of a system data store
• FIG.4 is a method for intercepting vectorization data and a machine learning model prediction.
• FIG.5 is a method for detecting and responding to malicious acts directed towards a machine learning model.
• FIG.6 is a method for generating an alert.
• FIG.7 is an interface for reporting the status of detected malicious acts directed towards a machine learning model.
• FIG.8 is another interface for reporting the status of detected malicious acts directed towards a machine learning model.
• FIG.9 provides a computing environment for implementing the present technology.
DETAILED DESCRIPTION
• The present technology, roughly described, detects and responds to malicious acts directed towards machine learning models. Data fed into and output by a machine learning model is collected by a sensor. The data fed into the model includes vectorization data, which is generated from raw data provided from a requester, such as for example a stream of timeseries data. The output data may include a prediction or other output generated by the machine learning model in response to receiving the vectorization data. By receiving the vectorization data rather than the raw timeseries data, the privacy and context of the data is kept secret from the detection system that processes the vectorization data.
• The vectorization data and machine learning model output data are processed to determine whether the machine learning model is being subject to malicious act, such as for example an attack. The processing may include feeding the vectorization data and output into one or more of several machine learning models, hash based filtering, and correlation of input and output of other requesters. The processing results in a determination as to whether the current data is associated with the malicious act, if a trend suggests the data is not as predicted, or if a distributed attack is or has occurred. The output of the processing may indicate an attack score, for example in the form of a prediction whether the machine learning model is subject to malicious act. An alert may be generated based on the value of the attack score.
• A response for handling the request by a requester may be selected based on the output that includes the attack score, and a response may be generated and applied to the requestor. The response may be any of several responses, such as for example providing a false series of values, randomizing an output, implementing a honeypot response, or simply disconnecting the requester. The present system may also report status of the monitoring and malicious act detection trends of the machine learning model through one or more dashboards or interfaces.
• FIG.1 is a block diagram of a system for detecting and responding to malicious acts directed towards a machine learning model. The system ofFIG.1 includes users105,customer environment110,system environment130, andcustomers160. Customer environment includes atransformation module115 andmachine learning model125. In between the transformation module and machine learning model is adetection system sensor120.
• One or more users may provide a stream of data, such as a timeseries data, generalized input, or some other data type, totransformation module115. The transformation module may convert the received timeseries into a series of vectorized data. In some instances, the vectorized data may include an array of float numbers. The vectorization of the received data is then provided tomachine learning model125 for processing. After processing the vectorized data, machine learning model will provide an output, such as a prediction, intended for the requesting user105.
• Detection system sensor120 may collect the vectorized data provided bytransformation115 and as well as the output provided bymachine learning model125. Thesensor120 may then couple the vectorized data and model output, and transmit the coupled data to theprocessing engine145 ofsystem environment130.Sensor120 may forward the vectorization data received fromtransformation115 tomachine learning model125.Sensor120 may also provide the output ofmodel125 or implement a different response to the requesting user. For example,sensor120 may generate and transmit a response to the requesting user based on data received fromresponse engine155. In some instances,sensor120 may disconnect the requesting user based on response data received fromresponse engine155.
• The sensor may be implemented in several ways. In some instances, a sensor may be implemented as an API placed between the requesting user and the machine learning model. The API may intercept the request, and then send the request to the machine learning model as well as to a publisher API. The publisher API may then transmit the vectorization data to a processing engine. The sensor API may then receive the response generated by the customer's machine learning model, and forward the response to the requesting user, if no malicious act is detected, or generate a different response based on data received from theresponse engine155.
• In some instances, the sensor may be implemented by an API gateway as well as a proxy application. The API gateway may receive the request, provide the request to the proxy application, which may then forward the request to themachine learning model125 as well as a publisher. The publisher may then forward the request to the system environment for processing by theprocessing engine145. The machine learning model may provide a response to the proxy application, and the proxy application can also receive response data fromresponse engine155. The proxy application may then either forward the machine learning model response to the requesting user through the API gateway, if the user request is not associated with a malicious act, or may generate a response based on the response data received from theresponse engine155 when the request is associated with the malicious act on the machine learning model.
• In some instances, a vector traffic instance may be implemented to forward a received request to themachine learning model125. A traffic mirror source may collect the traffic originating from the vector traffic instance and provide the traffic to a traffic mirror target, which then provides the traffic to a network load balancer. The network load balancer may then forward the vectorization traffic data through a series of traffic mirror worker applications, which then forward the vectorization traffic toprocessing engine145. After processing the vectorization traffic,response engine155 may provide response data to the traffic mirror workers, which then generate a response to transmit to the vector traffic instance when a malicious act on the machine learning model is detected.
• Returning toFIG.1,system environment130 includes customer data store135,system data store140,processing engine145,alert engine150,response engine155,network application160, andcustomers165. Each ofcustomer environment110 andsystem environment130 may be implemented as one or more servers implementing the physical or logical modules115-125 and135-160 illustrated inFIG.1. In some instances, each environment is located in one or more cloud computing environments.
• Environments110 and130 may communicate over a network. In some instances, one or more modules may be implemented on separate machines in separate environments which may also communicate over a network. The network may be implemented by one or more networks suitable for communication between electronic devices, including but not limited to a local area network, wide-area networks, private networks, public network, wired network, a wireless network, a Wi-Fi network, an intranet, the Internet, a cellular network, a plain old telephone service, and any combination of these networks.
• The customer data store135 ofFIG.1 includes data associated with one or more customers. The customer data stored may be accessed by any module withinsystem environment130. More information for customer data135 is discussed with respect to the system ofFIG.2.
• System data140 includes data related tosystem environment130. System data may include event data, traffic data, timestamp data, and other data. The data may be accessed by any of modules145-160, and may be used to generate one or more dashboards for use bycustomers165. More details forsystem data store140 are discussed with respect toFIG.3.
• Processing engine145 may be implemented by one or more modules that receive and process coupled vectorization data and machine learning model output data. Processing the received coupled data may include applying one or more machine learning modeling techniques to the data to determine if a malicious act has been performed against the customer'smachine learning model125. The machine learning model techniques applied to the coupled data may include unsupervised learning or clustering, timeseries modeling, classification modeling, and other modeling techniques. After the coupled data has been processed, the processing engine generates an attack score and provides that score to alertengine150.Alert engine150 may generate an alert based on the value of the score. In some instances, different alerts may be provided based on the value of the score, with more urgent alerts generated for a higher score.Alert engine150 then passes the coupled data and the attack score toresponse engine155.Response engine155 may receive the attack score, and optionally other data, and select a response to implement with respect to the requestor that transmitted the request from which the vectorization data was created. The responses may include anything such as providing a false series of prediction values having a pattern of some sort, providing a randomized response, implementing a honeypot response, or disconnecting the requester. Information about the selected response is provided todetection system sensor120, which then generates and implements the response.
• Response engine provides the selected response and the attack score to networkapplication160.Network application160 may provide one or more APIs, integrations, or user interfaces, for example in the form of a dashboard, which may be accessed bycustomers165. The dashboard may provide information regarding any detected or suspected malicious acts, attack trends, statistics and metrics, and other data. Examples of dashboards providing malicious act data is discussed with respect toFIGS.7 and8.
• FIG.2 is a block diagram of a customer data store.Customer data store200 ofFIG.2 provides more detail of customer data store135 ofFIG.1. Customer data store may include customer data210. The customer data210 may include data associated with all customers that providemachine learning model125. Customer data may include, but is not limited to, a customer name, a unique user ID, a date that the customer data was created, a publisher token, a sensor identifier, and a letter identifier. The sensor identifier may indicate what sensor is associated with the customer'smachine learning model125 that is being monitored by the present system. A letter identifier may include an identifier for a particular alert engine that provides alert regarding the particular user'smachine learning model125.
• FIG.3 is a block diagram of a system data store.System data store300 ofFIG.3 provides more detail forsystem data store140 in the system ofFIG.1.System data store300 includesystem data310. Thesystem data310 may include, but is not limited to, vectorization data, prediction data, requester ID, sensor history, processing history, and alert history. The vectorization data may include the data generated bytransformation module115 with acustomer environment110, for each customer. Prediction data may include the output ofmachine learning model125 that is intercepted bysensor120, for each customer. Requester ID may include the source of raw data, such as timeseries data, which is provided from users105transformation115. Sensor history includes a log of the actions performed bysensor120, the platform on which sensors are implemented, and other data regarding each sensor. Processing history may include the history, such as log information, processing history, and other history forprocessing engine145 for each particular customer. Alert history includes data such as the events occurring fromalert engine150, the status ofalert engine150, and the alerts generated byalert engine150 for each particular customer.
• FIG.4 is a method for intercepting vectorization data and a machine learning model prediction. First, a customer environment receives a request consisting of raw data from a user requester atstep405. The raw data may include a stream of timeseries data, or other data provided directly from a requester to the customer's environment. Customer transformation engine then transforms the raw data into vectorization data atstep410. The vectorization data will not have any context associated with the requester, but will still be associated with a requester ID. The vectorization data cannot be processed to determine the identity of the requester. In some instances, the vectorization data may be in the format of an array of float numbers.
• The customer transformation engine transmits the vectorization data to a sensor atstep420. The sensor may be placed between thetransformation module115 and themachine learning model125 to collect, or in some cases intercept, vectorization data transmitted tomodel125.
• The sensor may be provided in a variety of formats. In some instances, the sensor may be provided as an API to which vectorization data can be directed. In some instances, the sensor may be implemented as a network traffic capture tool that captures traffic intended for the machine learning model. In some instances, the sensor can be implemented using cloud libraries, for example a Python or C library, which can be plugged into customer software and used to direct vectorization traffic to theprocessing engine145.
• Machine learning model applies algorithms and/or processing to the vectorization data to generate a prediction atstep425.Machine learning model125 is part of the customer's environment and processes the vectorization data that is transmitted bytransformation module115. In some instances,sensor120, after collecting and/or intercepting the vectorization data, may forward the vectorization data tomachine learning model125 to be processed. Machine learning model then transmits the output prediction tosensor120 atstep430.
• The sensor couples the vectorization data and prediction atstep435. The sensor then transmits the coupled data to theremote system environment130 atstep440. At some point subsequently, the sensor receives response data based on the coupled data atstep445. The response data may be an indication of what response to send the requester generated by thesystem130. In particular, the response data may indicate a response selected byresponse engine155, other than the prediction output, to be provided to the requester based on a detection of a malicious act by the requester. The sensor generates a response based on the response data to the user requester atstep450. The response may be a pattern of data other than the output generated bymachine learning model125, randomized data, a honeypot based response, or a termination or disconnect of the session with the requester.
• FIG.5 is a method for detecting and responding to malicious acts directed towards a machine learning model. First, a processing engine receives the machine learning model vectorization and prediction coupled data from the sensor atstep505. Processing engine then analyzes the received data using one or more machine learning techniques to generate an attack score atstep510. Performing machine learning to analyze the data may include performing unsupervised learning or clustering on the received data, timeseries modeling, classification modeling, or some other machine learning based analysis and/or modeling on the coupled data. In some instances, analyzing the data includes performing clustering on similar data to determine if there is a distributed attack underway on the machine learning model atstep515. As a result of analyzing the coupled data, the processing engine generates an attack score. The attack score may be an indicator as to the likelihood or a predictor of whether themachine learning model125 provided by the customer is currently under attack or will be under attack in the near future.
• After determining the attack score, the processing engine provides the coupled data and the attack score to an alert engine atstep520. In some instances, thealert engine150 ofFIG.1 may have several instances, with one instance per customer. The alert engine receives a coupled data and attack scores, generates alert as needed based on the received data and the scores, and provides the scores and data to a response engine atstep525. The alert engine may generate a different alert based on the value of the received data. More details for an alert engine generating an alert is discussed with respect to the method ofFIG.6.
• A response engine receives the coupled data and the attack score and generates response data for the user requester atstep530. The response data may include a selected response to apply to the requester if the attack score is over a certain threshold. For example, if the attack scores over 50%, then a response other than the output generated bymachine learning model125 may be provided to the requester that provided the raw data totransformation115. The selected response may be based on user request, the category of the malicious act, the time or date of the response, the history of attacks from the particular requestor, and other data. The response engine transmits the response data for the user requester to the sensor atstep535. The sensor receives the response data and generates a response based on the response data atstep540. The sensor executes the response, in some cases transmitting a response to the user requester, based on the received response data atstep545. The status of any attack on the machine learning model owned by the customer can be reported atstep550. The reporting may include details, including raw data, metrics, current status, regarding the monitoring and detection data for sharinglearning model125. More details regarding dashboards that provide reporting data to a customer are discussed with respect toFIGS.7 and8.
• FIG.6 is a method for generating an alert. The method ofFIG.6 provides more detail forstep525 of the method ofFIG.5.Alert engine150 receives coupled data and attack scores fromprocessing engine145 atstep605. A determination is made as to whether the attack score satisfies a highest threshold atstep610. The highest threshold may indicate whether the highest level of alert should be generated for the particular vectorization data and output coupled pair. In some instances, an attack score of 90% or higher would satisfy the highest threshold. If the attack score does not satisfy the highest threshold, the method ofFIG.6 continues to step620. If the attack scores to satisfy the highest threshold, a hilar flag is generated based on the attack scores, and the other data is stored insystem data140.
• A determination is made as to whether the attack score satisfies the second highest threshold atstep620. In some instances, a second highest threshold may be between 80% to 90%. If the attack scores to satisfy the second highest threshold, a medium alert flag may be generated atstep625 based on the attack scores, and the medium alert flag may be stored insystem data store140.
• A determination is made as to whether the attack score satisfies a third highest threshold atstep630. If the attack score does satisfy the third highest threshold, a low alert flag is generated based on the attack scores, and the alert is stored insystem data store140. If the attack score does not satisfy the third highest threshold, no alert flag is generated for the attack score. In this case,detection system sensor120 may provide the generated output of themachine learning model125 to the requester that provided the original request.
• FIG.7 is an interface for reporting the status of detected malicious acts directed towards a machine learning model. The interface ofFIG.7 includes windows for reporting events, risk, category, sensors, requesters, events, and event details. The events window may indicate the number of events, the number of predictions, and the number of evasions recorded by the present system. The risk window can report the number of high risks, medium risks, and low risks. The high, medium, and low levels may be associated with the alert levels provided byalert engine150. The category window may report categories of attacks on a customer's machine learning model. In theinterface700 the categories include inference categories, replication categories, and division categories. The sensors window indicates the number of sensors dedicated to a user's machine learning model, and the number of requesters that have been requesting usage of the machine learning model. The events window indicates the total number of events that occurred with respect to the machine learning model of the customer.
• The event details window indicates an event identifier, timestamp, category of attack, risk level, and context for each particular event. The event details may be scrolled and searched withininterface700.
• FIG.8 is another interface for reporting the status of detected malicious acts directed towards a machine learning model. Interface800 ofFIG.8 provides additional data regarding the monitoring and threat detection associated withmachine learning model125. In particular, theinterface800 ofFIG.8 may provide data in graphical format related to a mean vector score, asset deployment, risk tolerance, and geographical information regarding the core of the requesters using themachine learning model125. The geographical information may be split into zones, and the data provided may include zone usage, response distribution for zone, and a regional view of the zones.Interface800 may also graphically display an anomaly distribution, asset health, number of active deployments, and anomalies by day, week, or month. Additionally,interface800 may illustrate risk distribution in different formats, such as a bar view or calendar view.
• The interfaces ofFIGS.7 and8 are exemplary, and are not intended to be limiting. The data collected, monitored, and the actions taken may be reported as raw data, metrics may be generated from the data, and trends may be determined from the data, all of which may be reported through one or more interfaces or dashboards.
• FIG.9 is a block diagram of a computing environment for implementing the present technology.System900 ofFIG.9 may be implemented in the contexts of the likes of machines that implement detection system sensory120,data stores135 and140,processing engine145,alert engine150,response engine155, andnetwork application160. Thecomputing system900 ofFIG.9 includes one ormore processors910 andmemory920.Main memory920 stores, in part, instructions and data for execution byprocessor910.Main memory920 can store the executable code when in operation. Thesystem900 ofFIG.9 further includes amass storage device930, portable storage medium drive(s)940,output devices950,user input devices960, agraphics display970, andperipheral devices980.
• The components shown inFIG.9 are depicted as being connected via asingle bus990. However, the components may be connected through one or more data transport means. For example,processor unit910 andmain memory920 may be connected via a local microprocessor bus, and themass storage device930, peripheral device(s)980,portable storage device940, anddisplay system970 may be connected via one or more input/output (I/O) buses.
• Mass storage device930, which may be implemented with a magnetic disk drive, an optical disk drive, a flash drive, or other device, is a non-volatile storage device for storing data and instructions for use byprocessor unit910.Mass storage device930 can store the system software for implementing embodiments of the present invention for purposes of loading that software intomain memory920.
• Portable storage device940 operates in conjunction with a portable non-volatile storage medium, such as a floppy disk, compact disk or Digital video disc, USB drive, memory card or stick, or other portable or removable memory, to input and output data and code to and from thecomputer system900 ofFIG.9. The system software for implementing embodiments of the present invention may be stored on such a portable medium and input to thecomputer system900 via theportable storage device940.
• Input devices960 provide a portion of a user interface.Input devices960 may include an alpha-numeric keypad, such as a keyboard, for inputting alpha-numeric and other information, a pointing device such as a mouse, a trackball, stylus, cursor direction keys, microphone, touch-screen, accelerometer, and other input devices. Additionally, thesystem900 as shown in FIG.9 includesoutput devices950. Examples of suitable output devices include speakers, printers, network interfaces, and monitors.
• Display system970 may include a liquid crystal display (LCD) or other suitable display device.Display system970 receives textual and graphical information and processes the information for output to the display device.Display system970 may also receive input as a touch-screen.
• Peripherals980 may include any type of computer support device to add additional functionality to the computer system. For example, peripheral device(s)980 may include a modem or a router, printer, and other device.
• The system of900 may also include, in some implementations, antennas, radio transmitters andradio receivers990. The antennas and radios may be implemented in devices such as smart phones, tablets, and other devices that may communicate wirelessly. The one or more antennas may operate at one or more radio frequencies suitable to send and receive data over cellular networks, Wi-Fi networks, commercial device networks such as a Bluetooth device, and other radio frequency networks. The devices may include one or more radio transmitters and receivers for processing signals sent and received using the antennas.
• The components contained in thecomputer system900 ofFIG.9 are those typically found in computer systems that may be suitable for use with embodiments of the present invention and are intended to represent a broad category of such computer components that are well known in the art. Thus, thecomputer system900 ofFIG.9 can be a personal computer, handheld computing device, smart phone, mobile computing device, workstation, server, minicomputer, mainframe computer, or any other computing device. The computer can also include different bus configurations, networked platforms, multi-processor platforms, etc. Various operating systems can be used including Unix, Linux, Windows, Macintosh OS, Android, as well as languages including Java, .NET, C, C++, Node.JS, and other suitable languages.
• The foregoing detailed description of the technology herein has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments were chosen to best explain the principles of the technology and its practical application to thereby enable others skilled in the art to best utilize the technology in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the technology be defined by the claims appended hereto.

Claims (20)

What is claimed is:

1. A method for monitoring a machine learning-based system for malicious acts, comprising:

receiving vectorization data by a sensor a server, the vectorization data derived from input data intended for a first machine learning model and provided by a requestor;

receiving, by the sensor, an output generated by the machine learning model, the machine learning model generating the output in response to receiving the vectorization data;

transmitting the vectorization data and the output to a processing engine by the sensor;

processing the vectorization data and the output by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data; and

applying a response to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.

2. The method ofclaim 1, wherein applying the response includes selecting, by a response engine, a response based on an output by a second machine learning model within the processing engine, the output of the second machine learning model including a prediction of an attack on the first machine learning model.

3. The method ofclaim 1, further comprising collecting the vectorization data by a sensor component, the sensor component transmitting the collected vectorization data to the processing engine on the server.

4. The method ofclaim 3, wherein the sensor component is created in a computing environment that proxies the first machine learning model.

5. The method ofclaim 3, further including:

collecting the output generated by the first machine learning model by the sensor component;

coupling the vectorization data and output by the sensor component; and

transmitting the coupled vectorization data and output to the processing engine by the sensor component.

6. The method ofclaim 3, further including:

intercepting the output of the first machine learning model by a sensor component; and

transmitting a response generated by the sensor to the requestor in place of the output, the response generated based at least in part on the attack score.

7. The method ofclaim 1, further comprising generating an alert based on the attack score.

8. The method ofclaim 1, further comprising reporting attack data to a user through a graphical interface, the attack data based at least in part on the attack score.

9. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for monitoring a machine learning-based system for malicious acts, the method comprising:

receiving vectorization data by a sensor, the vectorization data derived from input data intended for a first machine learning model and provided by a requestor;

receiving, by the sensor, an output generated by the machine learning model, the machine learning model generating the output in response to receiving the vectorization data;

transmitting the vectorization data and the output to a processing engine by the sensor;

processing the vectorization data and the output by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data; and

applying a response to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.

10. The non-transitory computer readable storage medium ofclaim 9, wherein applying the response includes selecting, by a response engine, a response based on an output by a second machine learning model within the processing engine, the output of the second machine learning model including a prediction of an attack on the first machine learning model.

11. The non-transitory computer readable storage medium ofclaim 9, the method further comprising collecting the vectorization data by a sensor component, the sensor component transmitting the collected vectorization data to the processing engine on the server.

12. The non-transitory computer readable storage medium ofclaim 11, wherein the sensor component is created in a computing environment that implements the first machine learning model.

13. The non-transitory computer readable storage medium ofclaim 11, the method further including:

collecting the output generated by the first machine learning model by the sensor component;

coupling the vectorization data and output by the sensor component; and

transmitting the coupled vectorization data and output to the processing engine by the sensor component.

14. The non-transitory computer readable storage medium ofclaim 11, the method further including:

intercepting the output of the first machine learning model by a sensor component; and

transmitting a response generated by the sensor to the requestor in place of the output, the response generated based at least in part on the attack score.

15. The non-transitory computer readable storage medium ofclaim 9, the method further comprising generating an alert based on the attack score.

16. The non-transitory computer readable storage medium ofclaim 9, the method further comprising reporting attack data to a user through a graphical interface, the attack data based at least in part on the attack score.

17. A system for monitoring a machine learning-based system for malicious acts, comprising:

one or more servers including a memory and a processor; and

one or more modules stored in the memory and executed by the processor to receive vectorization data, by sensor, the vectorization data derived from input data intended for a first machine learning model and provided by a requestor, receive, by the sensor, an output generated by the machine learning model, the machine learning model generating the output in response to receiving the vectorization data, transmit the vectorization data and the output to a processing engine by the sensor, process the vectorization data and the output by the processing engine to generate an attack score, the attack score indicating a likelihood of a malicious action towards the machine learning model via the vectorization data, and apply a response to a request associated with the requestor, the response based at least in part on the attack score, the response applied in place of the output of the first machine learning model.

18. The system ofclaim 17, wherein applying the response includes selecting, by a response engine, a response based on an output by a second machine learning model within the processing engine, the output of the second machine learning model including a prediction of an attack on the first machine learning model.

19. The system ofclaim 17, the modules further executable to collect the vectorization data by a sensor component, the sensor component transmitting the collected vectorization data to the processing engine on the server.

20. The system ofclaim 19, wherein the sensor component is created in a computing environment that implements the first machine learning model.

Images