WebRTCWindowSystem is a BitmapWindowSystem with a WebRTC server that sends video frames to remote clients for visualization. More...

#include <WebRTCWindowSystem.h>

Inheritance diagram for cloudViewer::visualization::webrtc_server::WebRTCWindowSystem:

[legend]

Collaboration diagram for cloudViewer::visualization::webrtc_server::WebRTCWindowSystem:

[legend]

Classes
struct	Impl

Public Member Functions
virtual	~WebRTCWindowSystem ()

OSWindow	CreateOSWindow (gui::Window o3d_window, int width, int height, const char title, int flags) override

void	DestroyWindow (OSWindow w) override

std::vector< std::string >	GetWindowUIDs () const
	List available windows. More...

std::string	GetWindowUID (OSWindow w) const

OSWindow	GetOSWindowByUID (const std::string &uid) const

void	StartWebRTCServer ()
	Start WebRTC server in a background thread. More...

std::string	OnDataChannelMessage (const std::string &message)

void	RegisterDataChannelMessageCallback (const std::string &class_name, const std::function< std::string(const std::string &)> callback)

void	OnFrame (const std::string &window_uid, const std::shared_ptr< core::Tensor > &im)
	Server -> client frame. More...

void	SendInitFrames (const std::string &window_uid)

std::string	CallHttpAPI (const std::string &entry_point, const std::string &query_string="", const std::string &data="") const
	Call PeerConnectionManager's web request API. More...

void	EnableWebRTC ()

void	DisableHttpHandshake ()

void	CloseWindowConnections (const std::string &window_uid)
	Close all WebRTC connections that correspond to a Window. More...

Public Member Functions inherited from cloudViewer::visualization::gui::BitmapWindowSystem
	BitmapWindowSystem (Rendering mode=Rendering::NORMAL)

	~BitmapWindowSystem ()

void	Initialize () override

void	Uninitialize () override

void	SetOnWindowDraw (OnDrawCallback callback)

void	WaitEventsTimeout (double timeout_secs) override

Size	GetScreenSize (OSWindow w) override

void	PostRedrawEvent (OSWindow w) override

void	PostMouseEvent (OSWindow w, const MouseEvent &e)

void	PostKeyEvent (OSWindow w, const KeyEvent &e)

void	PostTextInputEvent (OSWindow w, const TextInputEvent &e)

bool	GetWindowIsVisible (OSWindow w) const override

void	ShowWindow (OSWindow w, bool show) override

void	RaiseWindowToTop (OSWindow w) override

bool	IsActiveWindow (OSWindow w) const override

Point	GetWindowPos (OSWindow w) const override

void	SetWindowPos (OSWindow w, int x, int y) override

Size	GetWindowSize (OSWindow w) const override

void	SetWindowSize (OSWindow w, int width, int height) override

Size	GetWindowSizePixels (OSWindow w) const override

void	SetWindowSizePixels (OSWindow w, const Size &size) override

float	GetWindowScaleFactor (OSWindow w) const override

float	GetUIScaleFactor (OSWindow w) const override

void	SetWindowTitle (OSWindow w, const char *title) override

Point	GetMousePosInWindow (OSWindow w) const override

int	GetMouseButtons (OSWindow w) const override

void	CancelUserClose (OSWindow w) override

void *	GetNativeDrawable (OSWindow w) override

rendering::FilamentRenderer *	CreateRenderer (OSWindow w) override

void	ResizeRenderer (OSWindow w, rendering::FilamentRenderer *renderer) override

MenuBase *	CreateOSMenu () override

Public Member Functions inherited from cloudViewer::visualization::gui::WindowSystem
virtual	~WindowSystem ()

Static Public Member Functions
static std::shared_ptr< WebRTCWindowSystem >	GetInstance ()

Additional Inherited Members
Public Types inherited from cloudViewer::visualization::gui::BitmapWindowSystem
enum class	Rendering { NORMAL , HEADLESS }

using	OnDrawCallback = std::function< void(Window *, std::shared_ptr< core::Tensor >)>

Public Types inherited from cloudViewer::visualization::gui::WindowSystem
using	OSWindow = void *

Static Public Attributes inherited from cloudViewer::visualization::gui::WindowSystem
static constexpr int	FLAG_VISIBLE = 0

static constexpr int	FLAG_HIDDEN = (1 << 0)

static constexpr int	FLAG_TOPMOST = (1 << 1)

Detailed Description

WebRTCWindowSystem is a BitmapWindowSystem with a WebRTC server that sends video frames to remote clients for visualization.

WebRTCWindowSystem owns a PeerConnectionManager, which manages all things related to the WebRTC connections, e.g. get media lists, get and add ICE candidates, connect to a media and hangup.

When the client visit a CloudViewer visualizer's website for visualization (a.k.a. standalone mode), an HTTP handshake server will be used to serve the website and perform handshake to establish the WebRTC connection. In Jupyter mode, the HTTP handshake server is disabled and the handshake is done via Jupyter's JavaScript<->Python communication channel.

WebRTCWindowSystem shall be used as a global singleton. Both the PeerConnectionManager and the HTTP handshake server runs on different threads.

Definition at line 37 of file WebRTCWindowSystem.h.

Constructor & Destructor Documentation

◆ ~WebRTCWindowSystem()

cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::~WebRTCWindowSystem ( )

virtual

Definition at line 199 of file WebRTCWindowSystem.cpp.

Member Function Documentation

◆ CallHttpAPI()

std::string cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::CallHttpAPI	(	const std::string &	entry_point,
		const std::string &	query_string = `""`,
		const std::string &	data = `""`
	)		const

Call PeerConnectionManager's web request API.

This function is called in JavaScript via Python binding to mimic the behavior of sending HTTP request via fetch() in JavaScript.

With fetch: data = {method: "POST", body: JSON.stringify(candidate)}; fetch("/api/addIceCandidate?peerid=" + peerid, data);

Now with CallHttpAPI: cloudViewer.visualization.webrtc_server("/api/addIceCandidate", "?peerid=" + peerid, data["body"]);

Parameters

entry_point	URL part before '?'.
query_string	URL part after '?', including '?'. If '?' is not the first character or if the stirng is empty, the query_string is ignored.
data	JSON-encoded string.

Definition at line 421 of file WebRTCWindowSystem.cpp.

References cloudViewer::utility::JsonToString(), LogDebug, result, and cloudViewer::utility::StringToJson().

◆ CloseWindowConnections()

void cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::CloseWindowConnections ( const std::string & window_uid )

Close all WebRTC connections that correspond to a Window.

Definition at line 497 of file WebRTCWindowSystem.cpp.

Referenced by DestroyWindow().

◆ CreateOSWindow()

WebRTCWindowSystem::OSWindow cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::CreateOSWindow	(	gui::Window *	o3d_window,
		int	width,
		int	height,
		const char *	title,
		int	flags
	)

overridevirtual

Reimplemented from cloudViewer::visualization::gui::BitmapWindowSystem.

Definition at line 204 of file WebRTCWindowSystem.cpp.

References height, LogInfo, o3d_window, StartWebRTCServer(), and width.

◆ DestroyWindow()

void cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::DestroyWindow ( OSWindow w )

overridevirtual

Reimplemented from cloudViewer::visualization::gui::BitmapWindowSystem.

Definition at line 220 of file WebRTCWindowSystem.cpp.

References CloseWindowConnections(), and LogInfo.

◆ DisableHttpHandshake()

void cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::DisableHttpHandshake ( )

HTTP handshake server is enabled by default. In Jupyter environment, call DisableHttpHandshake() before StartWebRTCServer().

Definition at line 492 of file WebRTCWindowSystem.cpp.

References LogInfo.

◆ EnableWebRTC()

void cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::EnableWebRTC ( )

Sets WebRTCWindowSystem as the default window system in Application. This enables a global WebRTC server and each gui::Window will be rendered to a WebRTC video stream.

Definition at line 487 of file WebRTCWindowSystem.cpp.

References cloudViewer::visualization::gui::Application::GetInstance(), GetInstance(), LogInfo, and cloudViewer::visualization::gui::Application::SetWindowSystem().

◆ GetInstance()

std::shared_ptr< WebRTCWindowSystem > cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::GetInstance ( )

static

Definition at line 111 of file WebRTCWindowSystem.cpp.

Referenced by EnableWebRTC(), cloudViewer::visualization::webrtc_server::PeerConnectionManager::GetMediaList(), cloudViewer::visualization::webrtc_server::pybind_webrtc_server_functions(), and cloudViewer::visualization::webrtc_server::PeerConnectionManager::SendInitFramesToPeer().

◆ GetOSWindowByUID()

WebRTCWindowSystem::OSWindow cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::GetOSWindowByUID ( const std::string & uid ) const

Definition at line 245 of file WebRTCWindowSystem.cpp.

Referenced by OnDataChannelMessage(), and SendInitFrames().

◆ GetWindowUID()

std::string cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::GetWindowUID ( WebRTCWindowSystem::OSWindow w ) const

Definition at line 236 of file WebRTCWindowSystem.cpp.

◆ GetWindowUIDs()

std::vector< std::string > cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::GetWindowUIDs ( ) const

List available windows.

Definition at line 228 of file WebRTCWindowSystem.cpp.

◆ OnDataChannelMessage()

std::string cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::OnDataChannelMessage ( const std::string & message )

Client -> server message.

Returns: Message reply that should be sent back to the client.

Definition at line 357 of file WebRTCWindowSystem.cpp.

References format, GetOSWindowByUID(), LogDebug, LogInfo, cloudViewer::visualization::gui::BitmapWindowSystem::PostRedrawEvent(), and cloudViewer::utility::StringToJson().

◆ OnFrame()

void cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::OnFrame	(	const std::string &	window_uid,
		const std::shared_ptr< core::Tensor > &	im
	)

Server -> client frame.

Definition at line 402 of file WebRTCWindowSystem.cpp.

◆ RegisterDataChannelMessageCallback()

void cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::RegisterDataChannelMessageCallback	(	const std::string &	class_name,
		const std::function< std::string(const std::string &)>	callback
	)

When the data channel receives a valid JSON string, the class_name property of the JSON object will be examined and the corresponding callback function will be called. The callback should return a string reply, which will be sent back to the client.

Parameters

class_name	The value of the `class_name` property of the JSON object.
callback	The callback function that will be called when a JSON object with the matching `class_name` is received via the data channel.

// Register callback in C++
auto ws =
visualization::webrtc_server::WebRTCWindowSystem::GetInstance();
ws->EnableWebRTC();
ws->RegisterDataChannelMessageCallback("webapp/input", "input",
    [](const std::string &data) {
        std::string reply = fmt::format("Received dataChannel message"
        " for class_name 'webapp/input' with data: {}", data);
        utility::LogInfo(reply);
        return reply;
    });

   /* Send message in JavaScript to trigger callback. this is
*  WebRTCStreamer object */
   this.dataChannel.send('{"class_name":"webapp/input",
       "data":"Test event"}');

Definition at line 393 of file WebRTCWindowSystem.cpp.

References callback, and LogDebug.

◆ SendInitFrames()

void cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::SendInitFrames ( const std::string & window_uid )

Send initial frames. This flushes the WebRTC video stream. After the initial frames, new frames will only be sent at triggered events.

Definition at line 407 of file WebRTCWindowSystem.cpp.

References GetOSWindowByUID(), LogDebug, LogInfo, and cloudViewer::visualization::gui::BitmapWindowSystem::PostRedrawEvent().

◆ StartWebRTCServer()

void cloudViewer::visualization::webrtc_server::WebRTCWindowSystem::StartWebRTCServer ( )

Start WebRTC server in a background thread.

Definition at line 257 of file WebRTCWindowSystem.cpp.

References cloudViewer::visualization::webrtc_server::GetCustomSTUNServer(), cloudViewer::visualization::gui::Application::GetInstance(), LogError, LogInfo, cloudViewer::visualization::webrtc_server::s_cloudViewer_ice_servers, cloudViewer::visualization::webrtc_server::s_public_ice_servers, and cloudViewer::utility::SplitString().

Referenced by CreateOSWindow().

The documentation for this class was generated from the following files:

/root/ACloudViewer/libs/cloudViewer/visualization/webrtc_server/WebRTCWindowSystem.h
/root/ACloudViewer/libs/cloudViewer/visualization/webrtc_server/WebRTCWindowSystem.cpp

Classes

Public Member Functions

Static Public Member Functions

Additional Inherited Members