Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
forked fromvladmandic/human

Human: AI-powered 3D Face Detection & Rotation Tracking, Face Description & Recognition, Body Pose Tracking, 3D Hand & Finger Tracking, Iris Analysis, Age & Gender & Emotion Prediction, Gaze Tracking, Gesture Recognition

License

NotificationsYou must be signed in to change notification settings

CubFE/human

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Git VersionNPM VersionLast CommitLicenseGitHub Status ChecksVulnerabilities

Human Library

AI-powered 3D Face Detection & Rotation Tracking, Face Description & Recognition,
Body Pose Tracking, 3D Hand & Finger Tracking, Iris Analysis,
Age & Gender & Emotion Prediction, Gaze Tracking, Gesture Recognition, Body Segmentation


Highlights

  • Compatible with most server-side and client-side environments and frameworks
  • Combines multiple machine learning models which can be switched on-demand depending on the use-case
  • Related models are executed in an attention pipeline to provide details when needed
  • Optimized input pre-processing that can enhance image quality of any type of inputs
  • Detection of frame changes to trigger only required models for improved performance
  • Intelligent temporal interpolation to provide smooth results regardless of processing performance
  • Simple unified API
  • Built-in Image, Video and WebCam handling

Jump to Quick Start


Compatibility

  • Browser:
    Compatible with both desktop and mobile platforms
    Compatible withCPU,WebGL,WASM backends
    Compatible withWebWorker execution
    Compatible withWebView
  • NodeJS:
    Compatibile withWASM backend for executions on architectures wheretensorflow binaries are not available
    Compatible withtfjs-node using software execution viatensorflow shared libraries
    Compatible withtfjs-node using GPU-accelerated execution viatensorflow shared libraries and nVidia CUDA

Releases

Demos

Check outSimple Live Demo fully annotated app as a good start starting point (html)(code)

Check outMain Live Demo app for advanced processing of of webcam, video stream or images static images with all possible tunable options

  • To start video detection, simply pressPlay
  • To process images, simply drag & drop in your Browser window
  • Note: For optimal performance, select only models you'd like to use
  • Note: If you have modern GPU,WebGL (default) backend is preferred, otherwise selectWASM backend

Browser Demos

All browser demos are self-contained without any external dependencies

  • Full[Live][Details]: Main browser demo app that showcases all Human capabilities
  • Simple[Live][Details]: Simple demo in WebCam processing demo in TypeScript
  • Embedded[Live][Details]: Even simpler demo with tiny code embedded in HTML file
  • Face Detect[Live][Details]: Extract faces from images and processes details
  • Face Match[Live][Details]: Extract faces from images, calculates face descriptors and similarities and matches them to known database
  • Face ID[Live][Details]: Runs multiple checks to validate webcam input before performing face match to faces in IndexDB
  • Multi-thread[Live][Details]: Runs each Human module in a separate web worker for highest possible performance
  • NextJS[Live][Details]: Use Human with TypeScript, NextJS and ReactJS
  • ElectronJS[Details]: Use Human with TypeScript and ElectonJS to create standalone cross-platform apps
  • 3D Analysis with BabylonJS[Live][Details]: 3D tracking and visualization of heead, face, eye, body and hand
  • VRM Virtual Model Tracking with Three.JS[Live][Details]: VR model with head, face, eye, body and hand tracking
  • VRM Virtual Model Tracking with BabylonJS[Live][Details]: VR model with head, face, eye, body and hand tracking

NodeJS Demos

NodeJS demos may require extra dependencies which are used to decode inputs
See header of each demo to see its dependencies as they are not automatically installed withHuman

  • Main[Details]: Process images from files, folders or URLs using native methods
  • Canvas[Details]: Process image from file or URL and draw results to a new image file usingnode-canvas
  • Video[Details]: Processing of video input usingffmpeg
  • WebCam[Details]: Processing of webcam screenshots usingfswebcam
  • Events[Details]: Showcases usage ofHuman eventing to get notifications on processing
  • Similarity[Details]: Compares two input images for similarity of detected faces
  • Face Match[Details]: Parallel processing of facematch in multiple child worker threads
  • Multiple Workers[Details]: Runs multiple parallelhuman by dispaching them to pool of pre-created worker processes
  • Dynamic Load[Details]: Loads Human dynamically with multiple different desired backends

Project pages

Wiki pages

Additional notes


Seeissues anddiscussions for list of known limitations and planned enhancements

Suggestions are welcome!



App Examples

VisitExamples gallery for more examples


Options

All options as presented in the demo application...
demo/index.html


Results Browser:
[Demo -> Display -> Show Results ]


Advanced Examples

  1. Face Similarity Matching:
    Extracts all faces from provided input images,
    sorts them by similarity to selected face
    and optionally matches detected face with database of known people to guess their names

demo/facematch

  1. Face Detect:
    Extracts all detect faces from loaded images on-demand and highlights face details on a selected face

demo/facedetect

  1. Face ID:
    Performs validation check on a webcam input to detect a real face and matches it to known faces stored in database

demo/faceid


  1. 3D Rendering:

human-motion


  1. VR Model Tracking:

human-three-vrm
human-bjs-vrm

  1. Human as OS native application:

human-electron


468-Point Face Mesh Defails:
(view in full resolution to see keypoints)




Quick Start

Simply loadHuman (IIFE version) directly from a cloud CDN in your HTML file:
(pick one:jsdelirv,unpkg orcdnjs)

<!DOCTYPE HTML><scriptsrc="https://cdn.jsdelivr.net/npm/@vladmandic/human/dist/human.js"></script><scriptsrc="https://unpkg.dev/@vladmandic/human/dist/human.js"></script><scriptsrc="https://cdnjs.cloudflare.com/ajax/libs/human/3.0.0/human.js"></script>

For details, including how to useBrowser ESM version orNodeJS version ofHuman, seeInstallation


Code Examples

Simple app that uses Human to process video input and
draw output on screen using internal draw helper functions

// create instance of human with simple configuration using default valuesconstconfig={backend:'webgl'};consthuman=newHuman.Human(config);// select input HTMLVideoElement and output HTMLCanvasElement from pageconstinputVideo=document.getElementById('video-id');constoutputCanvas=document.getElementById('canvas-id');functiondetectVideo(){// perform processing using default configurationhuman.detect(inputVideo).then((result)=>{// result object will contain detected details// as well as the processed canvas itself// so lets first draw processed frame on canvashuman.draw.canvas(result.canvas,outputCanvas);// then draw results on the same canvashuman.draw.face(outputCanvas,result.face);human.draw.body(outputCanvas,result.body);human.draw.hand(outputCanvas,result.hand);human.draw.gesture(outputCanvas,result.gesture);// and loop immediate to the next framerequestAnimationFrame(detectVideo);returnresult;});}detectVideo();

or usingasync/await:

// create instance of human with simple configuration using default valuesconstconfig={backend:'webgl'};consthuman=newHuman(config);// create instance of HumanconstinputVideo=document.getElementById('video-id');constoutputCanvas=document.getElementById('canvas-id');asyncfunctiondetectVideo(){constresult=awaithuman.detect(inputVideo);// run detectionhuman.draw.all(outputCanvas,result);// draw all resultsrequestAnimationFrame(detectVideo);// run loop}detectVideo();// start loop

or usingEvents:

// create instance of human with simple configuration using default valuesconstconfig={backend:'webgl'};consthuman=newHuman(config);// create instance of HumanconstinputVideo=document.getElementById('video-id');constoutputCanvas=document.getElementById('canvas-id');human.events.addEventListener('detect',()=>{// event gets triggered when detect is completehuman.draw.all(outputCanvas,human.result);// draw all results});functiondetectVideo(){human.detect(inputVideo)// run detection.then(()=>requestAnimationFrame(detectVideo));// upon detect complete start processing of the next frame}detectVideo();// start loop

or using interpolated results for smooth video processing by separating detection and drawing loops:

consthuman=newHuman();// create instance of HumanconstinputVideo=document.getElementById('video-id');constoutputCanvas=document.getElementById('canvas-id');letresult;asyncfunctiondetectVideo(){result=awaithuman.detect(inputVideo);// run detectionrequestAnimationFrame(detectVideo);// run detect loop}asyncfunctiondrawVideo(){if(result){// check if result is availableconstinterpolated=human.next(result);// get smoothened result using last-known resultshuman.draw.all(outputCanvas,interpolated);// draw the frame}requestAnimationFrame(drawVideo);// run draw loop}detectVideo();// start detection loopdrawVideo();// start draw loop

or same, but using built-in full video processing instead of running manual frame-by-frame loop:

consthuman=newHuman();// create instance of HumanconstinputVideo=document.getElementById('video-id');constoutputCanvas=document.getElementById('canvas-id');asyncfunctiondrawResults(){constinterpolated=human.next();// get smoothened result using last-known resultshuman.draw.all(outputCanvas,interpolated);// draw the framerequestAnimationFrame(drawResults);// run draw loop}human.video(inputVideo);// start detection loop which continously updates resultsdrawResults();// start draw loop

or using built-in webcam helper methods that take care of video handling completely:

consthuman=newHuman();// create instance of HumanconstoutputCanvas=document.getElementById('canvas-id');asyncfunctiondrawResults(){constinterpolated=human.next();// get smoothened result using last-known resultshuman.draw.canvas(outputCanvas,human.webcam.element);// draw current webcam framehuman.draw.all(outputCanvas,interpolated);// draw the frame detectgion resultsrequestAnimationFrame(drawResults);// run draw loop}awaithuman.webcam.start({crop:true});human.video(human.webcam.element);// start detection loop which continously updates resultsdrawResults();// start draw loop

And for even better results, you can run detection in a separate web worker thread




Inputs

Human library can process all known input types:

  • Image,ImageData,ImageBitmap,Canvas,OffscreenCanvas,Tensor,
  • HTMLImageElement,HTMLCanvasElement,HTMLVideoElement,HTMLMediaElement

Additionally,HTMLVideoElement,HTMLMediaElement can be a standard<video> tag that links to:

  • WebCam on user's system
  • Any supported video type
    e.g..mp4,.avi, etc.
  • Additional video types supported viaHTML5 Media Source Extensions
    e.g.:HLS (HTTP Live Streaming) usinghls.js orDASH (Dynamic Adaptive Streaming over HTTP) usingdash.js
  • WebRTC media track using built-in support




Detailed Usage




TypeDefs

Human is written using TypeScript strong typing and ships with fullTypeDefs for all classes defined by the library bundled intypes/human.d.ts and enabled by default

Note: This does not include embeddedtfjs
If you want to use embeddedtfjs insideHuman (human.tf namespace) and still fulltypedefs, add this code:

import type * as tfjs from '@vladmandic/human/dist/tfjs.esm';
const tf = human.tf as typeof tfjs;

This is not enabled by default asHuman does not ship with fullTFJS TypeDefs due to size considerations
Enablingtfjs TypeDefs as above creates additional project (dev-only as only types are required) dependencies as defined in@vladmandic/human/dist/tfjs.esm.d.ts:

@tensorflow/tfjs-core, @tensorflow/tfjs-converter, @tensorflow/tfjs-backend-wasm, @tensorflow/tfjs-backend-webgl




Default models

Default models in Human library are:

  • Face Detection:MediaPipe BlazeFace Back variation
  • Face Mesh:MediaPipe FaceMesh
  • Face Iris Analysis:MediaPipe Iris
  • Face Description:HSE FaceRes
  • Emotion Detection:Oarriaga Emotion
  • Body Analysis:MoveNet Lightning variation
  • Hand Analysis:HandTrack & MediaPipe HandLandmarks
  • Body Segmentation:Google Selfie
  • Object Detection:CenterNet with MobileNet v3

Note that alternative models are provided and can be enabled via configuration
For example, body pose detection by default usesMoveNet Lightning, but can be switched toMultiNet Thunder for higher precision orMultinet MultiPose for multi-person detection or evenPoseNet,BlazePose orEfficientPose depending on the use case

For more info, seeConfiguration Details andList of Models




Diagnostics




Human library is written inTypeScript5.1 usingTensorFlow/JS4.10 and conforming to latestJavaScriptECMAScript version 2022 standard

Build target for distributables isJavaScriptEMCAScript version 2018


For details seeWiki Pages
andAPI Specification


StarsForksCode SizeCDN
DownloadsDownloadsDownloads

About

Human: AI-powered 3D Face Detection & Rotation Tracking, Face Description & Recognition, Body Pose Tracking, 3D Hand & Finger Tracking, Iris Analysis, Age & Gender & Emotion Prediction, Gaze Tracking, Gesture Recognition

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • HTML79.9%
  • TypeScript17.6%
  • JavaScript1.7%
  • Other0.8%

[8]ページ先頭

©2009-2025 Movatter.jp