Virtual Web Cameraを作ろう

Kishikawa Katsumi

April 27, 2020

Tweet Share

Transcript

1. Kishikawa Katsumi Virtual Web CameraΛ࡞Ζ͏ CoreMediaIO Device Abstraction Layer Plug-In

2. Snap Camera

3. None

4. Snap Camera Package Installer

5. Snap Camera Package Installer

6. Snap Camera Package Installer

7. None
8. None

9. CoreMediaIO DAL Plug-In

10. None

11. Sample Code • https://developer.apple.com/library/archive/samplecode/CoreMediaIO/Introduction/Intro.html • https://github.com/lvsti/CoreMediaIO-DAL-Example • https://github.com/johnboiles/obs-mac-virtualcam • https://github.com/johnboiles/coremediaio-dal-minimal-example

    • https://github.com/seanchas116/SimpleDALPlugin • https://github.com/kishikawakatsumi/VirtualCameraComposer-Example

12. Apple's Sample Code

13. None
14. None

15. johnboiles/obs-mac-virtualcam

16. johnboiles/coremediaio-dal-minimal-example

17. seanchas116/SimpleDALPlugin

18. Virtual Camera CoreMediaIO DAL Plug-In

19. CoreMediaIO DAL Plug-In

20. CoreMediaIO DAL Plug-In ݻఆͷIDɻCoreMediaIOͷϔομʹॻ͍ͯ͋Δɻ

21. CoreMediaIO DAL Plug-In ೚ҙͷIDɻuuidgenͳͲͰద౰ʹ࡞Δɻ ॳظԽ࣌ʹ౉͞ΕΔɻ

22. CoreMediaIO DAL Plug-In ΤϯτϦʔϙΠϯτɻ ಉ໊͡લͷؔ਺Λ༻ҙ͢Δɻ

23. CoreMediaIO DAL Plug-In #import <CoreMediaIO/CMIOHardwarePlugin.h> #import "PlugInInterface.h" #import "Logging.h" //!

    PlugInMain is the entrypoint for the plugin extern "C" { void* PlugInMain(CFAllocatorRef allocator, CFUUIDRef requestedTypeUUID) { DLogFunc(@""); if (!CFEqual(requestedTypeUUID, kCMIOHardwarePlugInTypeID)) { return 0; } return PlugInRef(); } }

24. CoreMediaIO DAL Plug-In #import <CoreMediaIO/CMIOHardwarePlugin.h> #import "PlugInInterface.h" #import "Logging.h" //!

    PlugInMain is the entrypoint for the plugin extern "C" { void* PlugInMain(CFAllocatorRef allocator, CFUUIDRef requestedTypeUUID) { DLogFunc(@""); if (!CFEqual(requestedTypeUUID, kCMIOHardwarePlugInTypeID)) { return 0; } return PlugInRef(); } } C++Ͱॻ͘ͳΒϚϯάϦϯά͞Εͳ͍Α͏ʹ͢Δɻ

25. CoreMediaIO DAL Plug-In import Foundation import CoreMediaIO @_cdecl("simpleDALPluginMain") func simpleDALPluginMain(allocator:

    CFAllocator, requestedTypeUUID: CFUUID) -> CMIOHardwarePlugInRef { NSLog("simpleDALPluginMain") return pluginRef } SwiftͰ΋ಉ͡ɻ

26. CoreMediaIO DAL Plug-In PlugIn *plugIn = [PlugIn SharedPlugIn]; plugIn.objectId =

    objectID; Device *device = [[Device alloc] init]; CMIOObjectID deviceId; error = CMIOObjectCreate(PlugInRef(), kCMIOObjectSystemObject, kCMIODeviceClassID, &deviceId); ... Stream *stream = [[Stream alloc] init]; CMIOObjectID streamId; error = CMIOObjectCreate(PlugInRef(), deviceId, kCMIOStreamClassID, &streamId); ... stream.objectId = streamId; [[ObjectStore SharedObjectStore] setObject:stream forObjectId:streamId]; device.streamId = streamId; // Tell the system about the Device error = CMIOObjectsPublishedAndDied(PlugInRef(), kCMIOObjectSystemObject, 1, &deviceId, 0, 0); ... // Tell the system about the Stream error = CMIOObjectsPublishedAndDied(PlugInRef(), deviceId, 1, &streamId, 0, 0); ... ΠχγϟϥΠζ͕ݺͼग़͞ΕͨΒ σόΠε΍ετϦʔϜΛొ࿥͢Δɻ ͜ΕͰΧϝϥσόΠεͱͯ͠બ୒ Ͱ͖ΔΑ͏ʹͳΔɻ

27. CoreMediaIO DAL Plug-In CMSampleBufferRef buffer; err = CMIOSampleBufferCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer,

    format, &timing, self.sequenceNumber, kCMIOSampleBufferNoDiscontinuities, &buffer ); if (err != noErr) { DLog(@"CMIOSampleBufferCreateForImageBuffer err %d", err); } CMSimpleQueueEnqueue(self.queue, buffer); // Inform the clients that the queue has been altered if (self.alteredProc != NULL) { (self.alteredProc)(self.objectId, buffer, self.alteredRefCon); } ͋ͱ͸αϯϓϧόοϑΝΛ Ͳ͏ʹ͔ͯ͠࡞ͬͯ Ωϡʔʹ௥Ճ͍͚ͯͩ͘͠ɻ

28. Development Tips

29. Development Tips UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ import Cocoa import AVFoundation class ViewController: NSViewController,

    AVCaptureVideoDataOutputSampleBufferDelegate { private let session = AVCaptureSession() override func viewDidLoad() { super.viewDidLoad() view.wantsLayer = true AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach { do { let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ if session.canAddInput(input) { session.addInput(input) } } catch { print(error) } } let previewLayer = AVCaptureVideoPreviewLayer() previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable] previewLayer.session = session if let layer = view.layer { previewLayer.frame = layer.bounds layer.addSublayer(previewLayer) } session.startRunning() } } σόοά΍ಈ࡞֬ೝΛ؆୯ʹ͢Δʹ͸ɾɾɾ

30. Development Tips UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ import Cocoa import AVFoundation class ViewController: NSViewController,

    AVCaptureVideoDataOutputSampleBufferDelegate { private let session = AVCaptureSession() override func viewDidLoad() { super.viewDidLoad() view.wantsLayer = true AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach { do { let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ if session.canAddInput(input) { session.addInput(input) } } catch { print(error) } } let previewLayer = AVCaptureVideoPreviewLayer() previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable] previewLayer.session = session if let layer = view.layer { previewLayer.frame = layer.bounds layer.addSublayer(previewLayer) } session.startRunning() } }

31. Development Tips UnknownͷAVCaptureDeviceͱͯ͠ݟ͑Δ import Cocoa import AVFoundation class ViewController: NSViewController,

    AVCaptureVideoDataOutputSampleBufferDelegate { private let session = AVCaptureSession() override func viewDidLoad() { super.viewDidLoad() view.wantsLayer = true AVCaptureDevice.DiscoverySession(deviceTypes: [.externalUnknown], mediaType: .video, position: .unspecified).devices.forEach { do { let input = try AVCaptureDeviceInput(device: $0) // ଞʹ΋ϓϥάΠϯ͕͋Δ৔߹͸ద౰ʹௐ੔͢Δ if session.canAddInput(input) { session.addInput(input) } } catch { print(error) } } let previewLayer = AVCaptureVideoPreviewLayer() previewLayer.autoresizingMask = [.layerWidthSizable, .layerHeightSizable] previewLayer.session = session if let layer = view.layer { previewLayer.frame = layer.bounds layer.addSublayer(previewLayer) } session.startRunning() } } ϓϥάΠϯ΍ΤΫεςϯγϣϯͱ͍ͬͨιϑτ΢ΣΞ͸ ͱʹ͔͘σόοά͕େม͕ͩɺ͜Ε͚ͩͰಉ͡ϓϩηεͰಈ͘ͷͰ ϩάͳͲ͕શ෦ݟ͑ΔɻPreviewLayerͰಈ࡞΋֬ೝͰ͖Δɻ ͨͩผͷΞϓϦͰಈ͔͢৔߹ͱڍಈ͕ҟͳΔ͜ͱ͕͋ΔͷͰద౓ʹ֬ೝ͢Δ

32. Development Tips αϯϓϧόοϑΝΛCIImageʹͯ͠ҐஔΛม͑ͯ߹੒ func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer,

    from connection: AVCaptureConnection) { if output == self.cameraCapture.output { guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } let cameraImage = CIImage(cvImageBuffer: imageBuffer) guard let screenImageBuffer = lastScreenImageBuffer else { return } let screenImage = CIImage(cvImageBuffer: screenImageBuffer) let translatedCameraImage = cameraImage.transformed(by: CGAffineTransform(translationX: screenImage.extent.width - cameraImage.extent.width, y: 0)) let compositedImage = translatedCameraImage.composited(over: screenImage) ... αϯϓϧόοϑΝ͸ͱʹ͔͘CIImageʹͯ͠͠·͏ͱɺૢ࡞͕؆୯ɻ ߹੒΍֦େॖখɾҠಈ΋CIImageͷϝιουΛݺͿ͚ͩɻ ͜Ε͸΢Πϯυ΢ͷΩϟϓνϟͱΧϝϥͷө૾Λ߹੒͍ͯ͠Δɻ

33. Development Tips αϯϓϧόοϑΝΛCIImageʹͯ͠ςΩετΛΦʔόʔϨΠ let windowImage = CGWindowListCreateImage(.null, .optionIncludingWindow, CGWindowID(windowID), [.bestResolution,

    .boundsIgnoreFraming]) { let ciImage = CIImage(cgImage: windowImage) if let text = self.settings["text"] as? String, !text.isEmpty { let bitmapImageRep = NSBitmapImageRep(ciImage: ciImage) let g = NSGraphicsContext(bitmapImageRep: bitmapImageRep) NSGraphicsContext.saveGraphicsState() NSGraphicsContext.current = g (text as NSString).draw(at: NSPoint(x: 200, y: ciImage.extent.height - 400), withAttributes: [.fon NSFont.boldSystemFont(ofSize: 400), .foregroundColor: NSColor.black]) NSGraphicsContext.restoreGraphicsState() αϯϓϧόοϑΝʹςΩετΛࡌͤΔͷ΋ CIImageʹͯ͠draw͢Ε͹OKɻ

34. Development Tips εΫϦʔϯશମͷΩϟϓνϟ class ScreenCapture: NSObject { private let session

    = AVCaptureSession() let output = AVCaptureVideoDataOutput() override init() { session.sessionPreset = .high if let input = AVCaptureScreenInput(displayID: CGMainDisplayID()) { if session.canAddInput(input) { session.addInput(input) if session.canAddOutput(output) { session.addOutput(output) } } } } func startRunning() { session.startRunning() } func stopRunning() { session.stopRunning() } } εΫϦʔϯશମͷΩϟϓνϟʹ͸ AVCaptureScreenInput͕༻ҙ͞Ε͍ͯΔɻ

35. Development Tips ΢Πϯυ΢ͷҰཡ if let windowList = CGWindowListCopyWindowInfo([.optionOnScreenOnly, .excludeDesktopElements], kCGNullWindowID)

    { arrayController.content = (windowList as NSArray).filter{ (entry) -> Bool in if let entry = entry as? NSDictionary, let sharingState = entry[kCGWindowSharingState] as? Int, sharingState != CGWindowSharingType.none.rawValue { return true } return false } ΢Πϯυ΢͝ͱͷεΫϦʔϯγϣοτΛࡱΔʹ͸ ·ͣ΢Πϯυ΢ͷҰཡ͔ΒIDΛऔಘͯ͠ɺ

36. Development Tips ΢Πϯυ΢ͷΩϟϓνϟ if let windowID = self.settings["windowID"] as? Int,

    let windowImage = CGWindowListCreateImage(.null, .optionIncludingWindow, CGWindowID(windowID), [.bestResolution, .boundsIgnoreFraming]) { let ciImage = CIImage(cgImage: windowImage) ઐ༻ͷؔ਺ʹ΢Πϯυ΢IDΛ౉͢ɻ

37. Development Tips Son of GrabʢGrabͱ͍͏Screenshot.appͷલ਎ͷΑ͏ͳΞϓϦͷαϯϓϧʣ https://developer.apple.com/library/archive/samplecode/SonOfGrab/ Introduction/Intro.html#//apple_ref/doc/uid/DTS10004490-Intro- DontLinkElementID_2 ΢Πϯυ΢͝ͱͷεΫϦʔϯγϣοτ͸ ͜ͷαϯϓϧίʔυ͕ศརɻ

    XIBʹΤϥʔ͕ग़ΔͷͰ։͍ͯߋ৽͕ඞཁʢࣗಈʣɻ

38. Development Tips σʔλͷड͚౉͠ʢϓϩηεؒ௨৴ʣ • File I/O • Distributed Notification •

    XPC Service • HTTPS (Maybe) • Unix IPC (Maybe) ઃఆΛม͑ͨΓ֎͔ΒσʔλΛ౉͍ͨ͠ͱ͍͏ͱ͖ɺ ϓϩηεؒ௨৴͕࢖͑Δʢ͸ͣʣɻ ͨͩFileܦ༝΍XPC Service͸ಈ͖·ͤΜͰͨ͠ɻ ΋ͱ΋ͱͷAppleͷαϯϓϧͰ͸Mach PortΛ࢖ͬͨ ϓϩηεؒ௨৴Λ͍ͯ͠ΔͷͰɺগͳ͘ͱ΋ ͦΕ͸࢖͑Δʢ͸ͣʣ