Tonido Platform: Developers Guide (DRAFT) Updated: June 29, 2009 Contents 1.
2.
3.
Introduction ...................................................................................................................................... 4 1.1.
What is Tonido? ........................................................................................................................ 7
1.2.
Why Tonido? ............................................................................................................................. 8
1.3.
Supported Platforms ................................................................................................................. 8
1.4.
Running Tonido ......................................................................................................................... 9
1.5.
What you need to know to build Tonido applications.............................................................. 9
1.6.
Reference Books ....................................................................................................................... 9
Build System .................................................................................................................................... 11 2.1.
What is included in the SDK .................................................................................................... 11
2.2.
Supported Platforms ............................................................................................................... 11
2.3.
Building on Windows .............................................................................................................. 11
2.4.
Building on Linux ..................................................................................................................... 12
2.5.
Building on Mac OSX ............................................................................................................... 12
Writing your first Tonido Application ............................................................................................. 13 3.1.
Name your application ............................................................................................................ 13
3.2.
Create a Application project ................................................................................................... 13
3.3.
Write one or more services..................................................................................................... 13
3.4.
Export Services from the shared library.................................................................................. 18
3.5.
Implement the required function calls ................................................................................... 18
3.6.
Writing a manifest................................................................................................................... 19
3.7.
Prepare Application icon files ................................................................................................. 19
3.8.
Create the application directory ............................................................................................. 20
3.9.
Restart Tonido ......................................................................................................................... 20
Tonido Platform Developers Guide © CodeLathe LLC
Page 1
4.
Tonido Platform Overview .............................................................................................................. 21 4.1.
System Components ............................................................................................................... 21
4.2.
The Tonido Runtime Platform ................................................................................................. 22
4.3.
Tonido Domain Server ............................................................................................................ 24
5.
Services and Messages Architecture .............................................................................................. 27 5.1.
Services ................................................................................................................................... 27
5.2.
Messages ................................................................................................................................. 29
5.3.
Variant Data ............................................................................................................................ 30
5.4.
Dispatcher ............................................................................................................................... 33
5.5.
Common Service Operations .................................................................................................. 33
5.6.
Deferred Dispatcher ................................................................................................................ 35
6.
Application Basics ........................................................................................................................... 36 6.1.
Tonido Directory Layout ......................................................................................................... 36
6.2.
Application Lifecycle ............................................................................................................... 39
6.3.
Manifests................................................................................................................................. 41
7.
HTTP Interface................................................................................................................................. 44 7.1.
Anatomy of a HTTP Request ................................................................................................... 45
7.2.
Anatomy of a HTTP Response ................................................................................................. 46
7.3.
HTTP Core Authentication ...................................................................................................... 47
8.
P2P Interface ................................................................................................................................... 48
9.
Authentication and Security ........................................................................................................... 52 9.1.
Tonido Security ....................................................................................................................... 52
9.2.
AuthKeys ................................................................................................................................. 52
10.
Static APIs ................................................................................................................................... 56
10.1.
Profile Manager APIs........................................................................................................... 56
10.2.
Email Manager APIs ............................................................................................................ 56
10.3.
Activity Manager APIs ......................................................................................................... 56
10.4.
Peer Manager APIs .............................................................................................................. 56
10.5.
Group Manager APIs ........................................................................................................... 56
10.6.
Mail Manager APIs .............................................................................................................. 56
10.7.
Network Manager APIs ....................................................................................................... 56
10.8.
Remote Access Manager APIs ............................................................................................. 56
Tonido Platform Developers Guide © CodeLathe LLC
Page 2
10.9.
Image Manipulation ............................................................................................................ 56
10.10.
Virtual FS ............................................................................................................................. 56
11.
Service APIs ................................................................................................................................. 57
11.1.
Timer ................................................................................................................................... 57
11.2.
Event Notifier Service.......................................................................................................... 57
11.3.
FileTransfer ......................................................................................................................... 58
11.4.
Network .............................................................................................................................. 58
11.5.
MachineInfo ........................................................................................................................ 58
11.6.
Sync Proxy ........................................................................................................................... 58
11.7.
Stats Service ........................................................................................................................ 58
12.
Guide to Building User Interfaces for Tonido Applications......................................................... 60
13.
Third party Libraries .................................................................................................................... 60
13.1.
POCO ................................................................................................................................... 60
13.2.
Boost ................................................................................................................................... 60
13.3.
GD........................................................................................................................................ 60
13.4.
Sqlite ................................................................................................................................... 60
13.5.
Crypto++ .............................................................................................................................. 60
13.6.
OpenSSL .............................................................................................................................. 60
13.7.
Lua ....................................................................................................................................... 60
13.8.
LuaBind................................................................................................................................ 60
14.
Patching System .......................................................................................................................... 61
15.
Listing your Plugin in Tonido App Store ...................................................................................... 62
16.
Recommendations, Tips and Tricks ............................................................................................ 63
16.1.
Running multiple Tonido instances on a single machine .................................................... 63
Tonido Platform Developers Guide © CodeLathe LLC
Page 3
1. Introduction Some History To make sense of the direction that software is going, one needs to study the evolution of computing. Evolution in living organisms happens due to environmental mechanisms that favor one form versus another (aka natural selection) as well as due to small random changes to existing forms that might get passed down from one generation to another (mutation). The analogy of software applications to living organisms breaks down at several levels including time (a few billion years versus about 50 years) but it still is a useful model to use. Early computers were monstrous in terms of size, cost and inefficiencies. And it was almost impossible for computers to exist without an army of attendants keeping it alive. Over time, major advances in technology, the invention of the transistor, improvements in manufacturing etc made computers affordable by big corporations, most Governments, then by medium businesses and finally by almost everyone. This was the Personal Computer revolution, which wrenched computing power from huge central servers and distributed it to the hands of the individual user. For the first time, a person could do what he wished with his machine without affecting anyone else. For the first time, he needn’t get a time slot for doing computing; the computing resources were his alone, waiting for him to use. This was the Personal Computer. The advent of the PC and Macs led to huge strides in computing. Users free to tinker with the hardware and software figured out innovative things to do as well as started demanding more and more from their machines; from dot matrix printers to color laser printouts, from command line interfaces to rich GUIs, from primitive pixilated games to full blown virtual worlds, the changes were tremendous. This evolution was so rapid that a PC about a couple of years old became obsolete and unable to run latest applications. The Internet However a small mutation appeared, that completely took PCs into a completely new direction.This is the advent of the Internet: The Information Super-Highway; The World Wide Web. A simple protocol over a copper wire (HTTP over TCP/IP) changed the history of Software Applications. The Internet started out humbly as a communication medium for people to send simple messages (email etc) to each other. With HTTP and Web browsers, it became a means for people to access, disseminate information effectively. The key driver behind the web was information, whether it is a recipe for a favorite dish or figuring out the reason the PC’s BIOS won’t POST. Because the information was so easy to access and could cross operating system walls unfettered, more and more people realized that the WWW can actually become a platform than just a conduit for information.
Tonido Platform Developers Guide © CodeLathe LLC
Page 4
The Browser is the new PC Along with this development came a lot of other economic changes which drove most desktop application developers out of business (except for the big software companies like Adobe etc). The factors were many, but one of them was the growing realization by software developers that building software applications for the desktop has lot more perils than building applications on the web. The perils included rampant piracy, almost impossible to administer software licensing, increasing competition from open source and free software and so on. On the other hand, building (good) web applications was (and is) a several orders of magnitude harder than desktop apps, but there was far less chance that what you build will become useless or pirated. Once consumers started using the application and invested some time and energy into it, they were bound to remain loyal, held captive by the immobility of the data. This sounded like a brilliant business model for many companies. To consumers, on the surface, this seemed good too. No more worrying about installing applications, just do it all from Internet applications. Upload all your files/photos/information into online services and access them anywhere a browser works. The allure was unmistakable. It is the scenario that we find ourselves in today in late 2008. AJAX lead the charge in promising the new internet. You know the old world is crumbling when Adobe previews a new Photoshop application running on the “Cloud”! So have we now come a full circle? We went from huge servers to completely independent standalone PCs and then back to huge server farms. For most people a day on the computer has become synonymous with using the browser. Reading email, news, writing a document on Google Docs etc. (A browser has become what Emacs used to be, the center of the computing universe). But Look Ma, I got all these cores! So most new applications are definitely headed towards the “cloud”(whatever that is). Almost all new exciting apps/ideas you hear about will somehow or the other use the HTTP protocol. Developers love it, consumers do too and what more do you want? That brings us to the next interesting question. What about all the horsepower we now run under the hood of our PCs? Core 2 Duos, Core 2 Quads. All pretty powerful, but pretty much useless if a browser is limited to making no more than 2 connections at a time to the web server and almost all the heavy lifting is done by a server on the server farm. But then the hardware companies assure you - “You DO need to buy the latest machine so you can surf 5x faster”. It cannot be farther away from the truth, since without changing your network connection for instance, you are not going to do anything significantly faster. I can attest to this personally as I am owner of a laptop that is more than 7 years old which can browse the Internet and be perfectly usable as my new screaming fast Core 2 Quad. Go figure! So where does this leave the computer hardware manufacturers?
Tonido Platform Developers Guide © CodeLathe LLC
Page 5
It is becoming harder to convince anyone to upgrade to the newest hardware. Also because CPU improvements have been increasing number of cores and not raw CPU speed you are going to see far less improvements in performance (Remember: Most software still doesn’t take advantage of multiple cores). You are probably better off upgrading your network and connectivity. And operating systems? Well they matter even less these days. Almost any operating system offers almost the identical experience if all you are seeing is the borders of Firefox all day. But what about Games? And so I believed too that games alone will stem the time against web applications. But here too things are changing. Not many standalone titles are being released these days on the PC. The consoles are where the action is. Of course, there will always be RPGs that can never be played on a console, but those are few and far between. The new direction towards some form of central server based applications has affected games as well. Look at the World of Warcraft and the other MMO games available. The game companies have finally figured out that online games are the perfect anti-piracy mechanisms. Are we done yet? So if all I have said seems to imply that the future of software applications is in the “cloud”, you might be right. That’s the future the big software businesses want to take the consumer towards. That’s the direction of the most profits, maximum control and complete lock-in. That’s the direction most of them are going toward. But there are a few of us out here who believe that this is not the direction that is the most beneficial over the long run for the average user. What you get from web applications, you lose in reduced freedom and the erosion of privacy. Trading up privacy and security of your data for the convenience of anywhere access doesn’t work. The scales don’t balance. Most people don’t see the threat yet or the incredible potential for misuse. What we need is something that balances the risks and rewards. We need applications that offer customers the accessibility of web applications with none of the risks. We want applications that can leverage all the computing horsepower to the max instead of rendering a paltry few pages every few minutes. We need applications without boundaries, which empower customers in the connected world. That might not be the future that businesses want, but that is the future that consumers will want. Tonido is a small step closer towards that future.
Tonido Platform Developers Guide © CodeLathe LLC
Page 6
1.1. What is Tonido?
Tonido is a P2P Personal Web application Platform Platform. Tonido is an extensible personal web application development platform platform,, that offers an array of applications to make your digital life better and safer. Since applications and data are always local, your private da data ta never has to reside on a third-party third server and is available to you anytime: both offline and online. Tonido applications are served right from your own desktop by the built built-in web server and allows easy access from any device with a web browser. In short, rt, your desktop has now become both a client and server. Tonido also connects to other Tonido instances directly via P2P communication through the Tonido Network. For example, the he Photos and Workspace apps use the Tonido network to directly synchronize photos and workspace data between members in a Tonido Group. Features • Total Privacy - User communication and information is neither monitored nor stored • Access from anywhere -Applications Applications and your data are accessible from any device with a web browser using your custom URL
Tonido Platform Developers Guide © CodeLathe LLC
Page 7
• • • • •
Available anytime-Access Tonido applications online or offline (i.e. with or without internet connectivity) Multiple Applications-Pick and choose your favorite applications Secure Communication-Encrypted data transfer Tonido Network-Private Invitation-only communities, Private and Public groups Multiple Operating Systems-Runs on Windows, Mac or Linux
1.2. Why Tonido? In the real world, relationships are private. We believe this should apply to your online world as well. We created Tonido and its applications to help you connect with people, who matter to you the most, while ensuring full protection and control of your data. With Tonido, you can now avoid uploading files and other digital assets to public online services. Access, communicate and collaborate directly without a data flowing through a thirdparty. Tonido helps you connect to other Tonido instances and resolves your custom URL, your data never passes through us. We firmly believe that a handful of companies controlling everybody's data is not healthy in the long-term and could lead to serious privacy issues and reduced freedoms. Instead, what we really need are applications with no boundaries that empower customers. Tonido is a small step towards accomplishing that. Why should you use Tonido? • • • •
If you don't want your personal information to be indexed by search engines... If you are concerned about the privacy and safety of your valuable data stored in online services... If you want to share your data with a chosen network of people securely and privately... If you would like to access your own files and other shared files from anywhere and everywhere...
1.3. Supported Platforms Tonido currently runs on the following Operating Systems: • • •
Windows XP, Windows Vista, Windows Home Server, Windows Server 2003 Mac OS X 10.5 and Higher Ubuntu Linux 8.04 and Higher
Tonido’s UI currently works on the following browsers • • •
Firefox 3.x and Higher IE 6 and Higher Safari 3 and Higher
Tonido Platform Developers Guide © CodeLathe LLC
Page 8
•
Google Chrome
1.4. Running Tonido You can download pre-built Tonido binaries from the Tonido website (http://www.tonido.com) for your operating system. Just install the binaries and open the webpage (http://127.0.0.1:10001). You will be able to create a Tonido profile and login into Tonido. 1.5. What you need to know to build Tonido applications Here’s a list of things you would need to know to build a Tonido application successfully. Note that this list will change as we add more scripting support and other language integration. Don’t worry too much if you don’t know some of these concepts, these are fairly simple to pick up and you won’t mostly need to worry too much about the gory details as these are abstracted away from you. • • • • •
Basic Multithreading concepts – Threads, Synchronization primitives HTTP GET / POST HTTP Cookies For extending existing HTTP APIs o Javascript/HTML or any browser based UI technology(See below) For new functionality o Concept of Shared Libraries o C++ Inheritance Abstract Classes Boost(or TR1) Shared Pointers
This guide is primarily focused on writing new functionality that does not use any existing HTTP APIs. If the application you are writing does not need any new HTTP APIs, then you will be able to write your plugin by simply re-using existing HTTP APIs. These APIs include calls for File Browsing, Downloading Files, Uploading Files and much more. If you want to have a UI for your application, you need to know either • HTML /Javascript libraries • Google GWT (Highly Recommended) • Adobe Flash • Microsoft Silverlight • Any other Web UI programming toolkits • You can even eschew the browser UI and use a Adobe AIR as your UI 1.6. Reference Books Here’s a list of books that will be useful to delve more into these subjects
Tonido Platform Developers Guide © CodeLathe LLC
Page 9
•
HTTP – The Definitive Guide by David Gourley and Brian Totty (http://www.amazon.com/HTTP-Definitive-Guide-DavidGourley/dp/1565925092/ref=sr_1_1?ie=UTF8&s=books&qid=1242401439&sr=8-1)
Tonido Platform Developers Guide © CodeLathe LLC
Page 10
2. Tonido Build System 2.1. What is included in the SDK • Pre-Compiled Thirdparty libraries for ease of compilation and setup • Tonido source code for the runtime • Class Reference for the runtime classes • Sample Applications o Dummy Application (Hello World Application) o Full Tonido Thots Application source, including GWT UI source o Developer Application source (Allows you to exercise some of the Tonido API easily) 2.2. Supported Platforms Tonido can be built on all three operating systems. • • •
Windows XP, Vista Ubuntu Linux 8.04 Mac OSX 10.5
2.3. Building on Windows 2.3.1. Requirements Tonido development requires the following tools on Windows. •
Compiler: Microsoft Visual Studio 2008(or SP1) or Microsoft VS Express C++ 2008(or SP1)
2.3.2. Setup For MS Visual Studio 2008 Users 1. Just install MS Visual Studio 2008 2. No other configuration is necessary For MS VS Express C++ 2008 Users 1. Download and install Microsoft Platform SDK (Version 6.1)
http://www.microsoft.com/downloads/details.aspx?familyid=E6E1C3DF-A74F4207-8586-711EBE331CDC&displaylang=en 2. Install Visual Studio 2008 Express C++ from http://www.microsoft.com/express/download/ Setting up Tonido SDK for development 1. Download Tonido Windows SDK Zip 2. Unzip the zip file into a directory (say TONIDOSDK) 3. Download Tonido Windows Thirdparty Zip 4. Unzip the zip file into the same directory (TONIDOSDK) 5. (Optional) Download and Unzip the Tonido Reference Class Documentation Tonido Platform Developers Guide © CodeLathe LLC
Page 11
After unzipping the files, the directories should look below TONIDOSDK +---------------------build +---------------------thirdparty +---------------------doc
6. Set P2P_BASE_PATH Environment Variable in Windows to that directory (TONIDOSDK). You will need to reboot 7. Run setup_Win32.bat inside the TONIDOSDK\thirdparty directory 8. Open the tonidosdk.sln file in visual studio in TONIDOSDK \build directory 9. Build Debug and Release 10. Go to the TONIDOSDK \build\bin\msvc\debug and run tonidoconsole, or set “tonidoconsole” project as the startup project in Visual Studio Solution Explorer and start project by hitting “F5” or Ctrl + F5. 11. Open http://127.0.0.1:10001 in your browser to see Tonido Viewing the Tonido SDK Class Reference The Tonido Platform classes are available as Doxygen style class reference documentation at TONIDOSDK\doc\reference\html\index.html If for some reason, you want to regenerate the class reference, you can run TONIDOSDK\doc\gendocsdk.bat file from the command line or double-click it form Window Explorer. 2.4. Building on Linux Coming soon 2.5. Building on Mac OSX Coming soon
Tonido Platform Developers Guide © CodeLathe LLC
Page 12
3. Writing your first Tonido Application Here’s a quick and handy guide to get started writing Tonido applications. This section is meant to give you a quick overview of the process on writing your first application. The rest of the guide goes through this in more detail. 3.1. Name your application First of all, you need to come up with a short name for your application. This will be used everywhere including URLs, folder names etc, so choose one that matches your application and does not conflict with existing applications (for e.g. torrent, photos, jukebox, workspace, webshare). For this example, let us use 3.2. Create a Application project This differs between windows and other UNIX based OSes. Essentially, your build project needs to be able to link with the Tonido Bootstrap SDK code and create a shared library (.dll in windows and .so in UNIX based OSes). The easiest way to get started is to copy the sample dummy project and change the settings to what you need. 3.3. Write one or more services Services are runtime components (C++ classes) that have well defined entry points and derive from the Tonido::Service abstract class. Most applications will usually have only one service. Ensure that you register your service and unregister your service and then setup your onMessage handler to handle messages that are sent to your service. Here’s the class declaration:
class DUMMYPLUGIN_API DummyPlugin: public Tonido::System::Service { public: //! Ctor /*! Details if needed */ DummyPlugin(); //! Dtor /*! Details if needed */ virtual ~DummyPlugin(); //! init /*! Inits the Service */ virtual bool a_basePath); //! /*!
init(const std::string&
name
Tonido Platform Developers Guide © CodeLathe LLC
Page 13
name of the service */ virtual const std::string&
name() const;
//! startup /*! Starts the Service */ virtual bool startup(Tonido::System::MessageDispatcher* dispatcher); //! /*!
shutdown
Shuts down the Service */ virtual bool //! /*!
shutdown();
onMessage OnMessage Virtual Override
*/ virtual void
onMessage ( const void* pSender, Message::SharedPtr& args
); private: Tonido::System::MessageDispatcher* std::string std::string };
m_pDispatcher; m_name; m_basePath;
We write a Dummy C++ class that derives from Tonido::System::Service. It implements the required functions (init, name, startup, shutdown, onMessage). Constructors and Destructors DummyPlugin::DummyPlugin() : m_pDispatcher(0), m_name("dummy") { } DummyPlugin::~DummyPlugin() { }
Tonido Platform Developers Guide © CodeLathe LLC
Page 14
Nothing particularly interesting in the constructors, we initialize our pointer variable that will hold a pointer to the Message Dispatcher. A message dispatcher is a singleton object which routes messages inside Tonido. The Message Dispatcher sends messages to Services by invoking the onMessage function. Message Dispatchers are explained more in depth in the following chapters. Service::Init Function bool DummyPlugin::init(const std::string& a_basePath) { m_basePath = a_basePath; return true; }
When the Dummy Application is inited, this function is invoked with the location of the application directory sent in the a_basePath variable. The application stores the location for possible future use in the m_basePath variable. Service::Name Function const std::string& DummyPlugin::name() const { return m_name; } The application will return its service name via the name function. Service::Startup Function bool DummyPlugin::startup(MessageDispatcher* dispatcher) { m_pDispatcher = dispatcher; /* Register our Service */ MessageEventInfo::SharedPtr pmi(new MessageEventInfo()); pmi->m_event += Poco::Delegate(this, &DummyPlugin::onMessage); bool retval = m_pDispatcher->registerMessageHandler(this, pmi); return true; }
This function is invoked when the service is being started. A pointer to the MessageDispatcher is sent in this function. The DummyService stores this pointer in the member variable m_pDispatcher.
Tonido Platform Developers Guide © CodeLathe LLC
Page 15
The next few lines register the onMessage handler function with the message dispatcher. This allows the message dispatcher to dispatch messages sent to this service ‘s name by invoking the onMessage handler of this Service object. The registration is done by creating a delegate to point to the DummyPlugin::onMessage function and then setting that delegate in the MessageEventInfo object and invoking the “registerMessageHandler” function in the dispatcher. Note that as soon as this function returns, the application service should be ready to process incoming messages via the onMessage handler. Service::Shutdown Function bool DummyPlugin::shutdown() { /* Unregister our Service */ return m_pDispatcher->unRegisterMessageHandler(this); } This function is invoked when the service is being shutdown (for example when Tonido is being shutdown). At this time, the service must unregister the message handler from the dispatcher. This is done by calling “unregisterMessageHandler” function. Service::onMessage Function void DummyPlugin::onMessage(const void* pSender, Message::SharedPtr& args) { std::string httpoutput; try { // ...Home Page httpoutput = "Dummy plugin" "Dummy Plugin Home" ""; Message::SharedPtr pSharedMessage = MessageUtil::createNewMessage(Defs::Service::SYNCPROXYSERVICE, Defs::Sync::MO_SYNC); pSharedMessage->setRawData(httpoutput.c_str(), httpoutput.length()); // ... Need to put the token in (*pSharedMessage)[Defs::Sync::MS_SYNC][Defs::Sync::S_SYNC_TOKEN] Tonido Platform Developers Guide © CodeLathe LLC
Page 16
= (*args.get())[Defs::Sync::MS_SYNC][Defs::Sync::S_SYNC_TOKEN]; MessageUtil::addHTTPResponse(pSharedMessage, Defs::HTTP::HTTP_REQ_OK, "text/html"); m_pDispatcher->postMessage(pSharedMessage); } catch(...) { // ... Exception Handling } } This is the primary function through which the service will receive Tonido Messages. A message is the primary means by which services are invoked as well as how they communicate. A message once created can be sent to another service either running in the same peer or in any other remote peer. Whenever a message is received by the service, this function will be invoked. Note that this function will be invoked in the thread context of the Message Dispatcher object. When this function is invoked, the message being sent to this service is sent via the arg variable. Since this dummy application is an example application, it simply displays a simple Hello World web page. So it doesn’t care about the actual contents of the message being sent, it simply creates an HTTP response by creating a HTML string and pushing it into the httpoutput variable and then sending it back to the HTTP Service for display in a browser. Note that this code assumes that a message is sent to this service via the HTTP service. The createNewMessage is an utility function that creates a new Tonido Message addressed to Defs::Service::SYNCPROXYSERVICE and the message operation is Defs::Sync::MO_SYNC. The service names and operations are constant strings for ease-of-use. We will talk more in depth about the SyncProxyService later, but suffice to say that the SyncProxy service acts as a proxy service for the HTTP service. If you need to send a HTTP response back to the browser for a given request, you will need to send to send it to the SyncProxy service. The response message payload is set to the httpoutput string value by using the setRawData function. Then to tie in the response with the request, we take the incoming token from the message and put it in the response message. The token is found in the message Variant Data Table. A Variant Data Table holds any number of generic data fields. The data in the data fields will depend upon the type of message and is completely service dependent.
Tonido Platform Developers Guide © CodeLathe LLC
Page 17
For example in the case of the SyncProxy service, when it sends a message to a service, it will always add a token string in the message variant data table at the following location. (*args)[Defs::Sync::MS_SYNC][Defs::Sync::S_SYNC_TOKEN] When sending back the response, we MUST send the token back so that the sync proxy service can figure out the corresponding request. We then set some more data fields required by the HTTP webserver that tells the server the content of the message as well as the HTTP status (OK for example) using the addHTTPResponse. Finally, the response message is sent back by invoking the postMessage function of the Message Dispatcher and sending it the response message. 3.4. Export Services from the shared library Every application must export one or more classes derived from Bootstrap’s Service class. To export your derived Services, you need to use POCO’s shared library Manifest macro as shown below. POCO_BEGIN_MANIFEST(Tonido::System::Service) POCO_EXPORT_CLASS (Tonido::Dummy::DummyPlugin) POCO_END_MANIFEST
Note even though you can have multiple services in your application, there SHOULD be one service whose registered name matches the name of the application (e.g. “dummy”). This is the main message handler for the application. 3.5. Implement the required function calls Every Application is expected to export a certain set of C Functions. These are invoked by Tonido when the plugin is loaded and unloaded. GetPluginDescription This provides a description of the application. GetPluginVersion This provides a plugin version. It is of the form of “MAJOR.MINOR.PATCH.BUILD” GetPluginName This provides a user viewable plugin name shown in the Core UI. It is REQUIRED to keep this name the same name as the directory in which this plugin is stored. GetPluginType This describes the type of plugin to the Tonido Plugin Loader. This should be “ApplicationPlugin”. GetPluginURL This is the relative URL required to open the UI for this plugin. If applications have a zip file where all HTML is located, they should specify the URL as “ui/PLUGINNAME/”. Tonido Platform Developers Guide © CodeLathe LLC
Page 18
pocoInitializeLibrary This function is called when the library is being loaded and any plugin load time initialization can be done here. pocoUninitializeLibrary This function is called when the library is being unloaded. 3.6. Writing a manifest Tonido primarily uses a manifest to figure out the application characteristics. If the manifest data is incorrect, Tonido will not load the application shared library. Here’s a simple manifest valid for Windows dummy http://www.tonido.com/app/dummy 0.0.0.1 \d+\.\d+\.\d+\.\d+ Windows NT IA32 CodeLathe Dummy Plugin Description false
3.7. Prepare Application icon files Every application needs two application icons that should be in the application folder. They should be in the gif format and should be named icon.gif and iconbg.gif. icon.gif iconbg.gif
Tonido Platform Developers Guide © CodeLathe LLC
25x25 icon shown in the application list in Tonido Admin (White Background) 25x25 icon shown in the application list in the left hand side on Tonido Admin (Gray background)
Page 19
3.8. Create the application directory Tonido requires the application files to be in a directory that is a subfolder in the Tonido “plugins” directory. Assume your application is named “dummy”, then you should create a folder called “dummy” and then have your shared library named dummy.dll (or dummy.so) inside that directory along with other application files. See below for sample layout. Tonido Base Directory +-------------plugins +--------dummy + icon.gif + iconbg.gif + dummy.dll(or .so) + manifest.xml
3.9. Restart Tonido When you restart Tonido, you will see your application automatically loaded by Tonido. If there is a problem loading a specific application, the information is put into the tonido.log file which is in the logs directory.
Tonido Platform Developers Guide © CodeLathe LLC
Page 20
4. Tonido Platform Overview 4.1. System Components There are two main software components in Tonido. The first one is the Tonido Runtime Platform. Platform You install this on your local computer and it contains and runs all the Tonido applications. The second one is the Tonido Domain Server or TDS for short. This is the component that is responsible for managing authentication, custom URL to IP address mapping and connection brokering. We will talk more about these components in depth in the coming sections.
Tonido Platform Developers Guide © CodeLathe LLC
Page 21
4.2. The Tonido Runtime Platform The Tonido runtime platform is installed on the end users machine. The installation consists of a runtime referred to as the Tonido Core runtime and Tonido applications. Tonido Core is the primary software component that provides programmable APIs and manages and provides HTTP and P2P connectivity. Tonido applications are shared libraries/dlls that are loaded/unloaded dynamically. Tonido has two primary interfaces to the outside world. • •
A HTTP interface which can be accessed by a browser A Tonido network interface which is used to connect to other Tonido instances directly
The primary function of the Tonido platform is to abstract these two interfaces from applications and present them uniformly as a single unified interface. Let’s look at how this is organized in more detail, as shown below.
HTTP Interface This functional block relates to the HTTP interface of Tonido. It manages HTTP client connections, credentials, loading UIs/Javascript using the Zip based Virtual File System. This is how users interact with Tonido since the user interface of Tonido is actually presented through the browser.
Tonido Platform Developers Guide © CodeLathe LLC
Page 22
Tonido P2P Network The Tonido network is a direct peer-to-peer based network, with a distributed login system. That simply means that Tonido instances will communicate with each other directly but identity will be managed by the TDS. The Tonido network functional block handles identity, connections with other Tonido instances, can transfer files and perform arbitrary data synchronization. Utility and Other APIs The Tonido Core runtime has other utility APIs that applications can use. They include service management, message routing, image manipulation, notifications, activity streams, management of Tonido Groups, loading, suspending and managing applications, Tonido updates, patching and notifications. Tonido Groups are collections of Tonido instances can be used as a logical block for certain operations by applications. For example, Tonido Photos will send a photo folder to a Tonido Group. Tonido Workspace uses Tonido Groups to synchronize workspace items. Tonido Services and Messages This is the glue that holds the entire Tonido architecture together. The Tonido platform comprises mainly of static APIs and dynamic Services. Static APIs are traditional APIs that can be invoked via a normal function call. These calls are synchronous, i.e. when you invoke them your code does not continue till the function call returns. Services, on the other hand cannot be invoked directly. If you need to interact with a service, you need to send messages to it. Unlike static API’s, messages are sent and processed asynchronously. i.e. your code continues to execute after sending a message. Once the service gets the message, it will perform an action and send you an optional response as another message. Messages are the primary unit of communication within Tonido and in-between Tonido instances. For example, if you want to send a response to the browser, you will send a message to the HTTP Service. If you want to initiate a file transfer to another Tonido instance, you send a message to the File Transfer Service requesting a file transfer. This allows Tonido applications to communicate with services in the local instance or remote Tonido instance the same way. When you want to write an application, you will actually write one or more services, compile it as a shared library and it becomes a Tonido Application. Browser Based UI Tonido uses the browser exclusively to provide interfaces to the applications. This provides a lot of flexibility, and allows users to access their applications from anywhere. Again, there are no restrictions on the type of UIs that applications have to implement, you can use plain HTML, AJAX, Flash, SilverLight or any of the hundreds of the other web development toolkits out there. The only
Tonido Platform Developers Guide © CodeLathe LLC
Page 23
requirement is that the browser UI communicates with the Tonido application using HTTP(S) (GET/POST) Security A fundamental part of the Tonido Platform is security. Significant amount of time has been spent on implementing secure communications, secure HTTP access and authentication methods. Since messages are primary means for communications inside and outside Tonido, there is a authorization key that every message carries. There is a associated privilege level (message from unknown Tonido contact, message from a Tonido Group contact etc) , a source level (for example, message from the UI, message from the network), and finally a location access level (message from local machine, message from trusted machine, message from remote machine ) etc that is part of the authorization key. Message handlers in applications are free to setup restrictions on levels required for a particular operation. We will discuss security in more detail in future chapters. Cross Platform and based on Open Frameworks Tonido runs natively in Windows, Linux, Mac OS X and therefore can fully use the underlying OS’s capabilities if available. Tonido also runs on i386 and ARM processors allowing Tonido to run on embedded and small footprint devices. Tonido does not reinvent the wheel when it comes to providing support for primitives like threading, shared pointers, image handling, storage and crypto. The Tonido platform integrates with the best-in-the-breed and permissive licensed libraries out there like boost, crypto++, sqlite and Poco to provide support. However, Tonido does not require application developers to stick to these libraries in their code, they are free to use any library out there. In fact, Tonido even provides support for applications to load and unload their own third party shared libraries. 4.3. Tonido Domain Server The other key part of the Tonido software ecosystem is the Tonido Domain Server (TDS). The TDS is designed to be an open system similar to email. Users can create accounts in any TDS run by anyone and communicate and collaborate with any other Tonido instance running on a different TDS. The TDS operates on well known IETF approved ports. 24465 for UDP and 24465 for TCP (HTTPS).
Tonido Platform Developers Guide © CodeLathe LLC
Page 24
See this concept below:
The Tonido Domain Server has the following functions. Authentication In Tonido the first fundamental requirement is the ability to identify a Tonido instance (or a device or user) uniquely. So the first time you install Tonido and open it up, you will have to create a new Tonido profile . This Tonido profile creates a Tonido ID for you on the Tonido Domain Server. Creating a Tonido ID is done via HTTPS connection to the TDS from the local Tonido Runtime. During profile creation, the TDS assigns a unique secret key to the Tonido Runtime for future use. No other information is transmitted. Also note that once the profile is created, Tonido does not require a connection with any online component including the TDS for further operation. URL to IP Address Mapping Not only does the profile guarantee uniqueness, it also provides you with a custom URL based on the domain the TDS is running on. For example, if the TDS runs on tonidoid.com, and if you create a profile named bob, you will now have your URL (bob.tonidoid.com) that will point to the IP address of your machine. This function is usually referred to as dynamic DNS. Connection Brokering
Tonido Platform Developers Guide © CodeLathe LLC
Page 25
The TDS serves as an intermediary to broker P2P connection setup between two Tonido instances. Normally, when Tonido runs, it is operating behind a router. In that case, it is not possible to make direct P2P connection to another Tonido instance unless the port is opened. TDS will notify the Tonido instance A when another Tonido instance B is attempting connection to A. Now A will start communication channel to B and B will meanwhile send communication to A via UDP. Most routers on seeing the two way traffic will assume that this is local initiated traffic and will allow the incoming traffic through. This establishes a direct P2P connection. This is referred to as a NAT punch. Utility Functions TDS also does provides utility functions like sending email, checking if your local Tonido URL is accessible correctly and so on.
Tonido Platform Developers Guide © CodeLathe LLC
Page 26
5. Services and Messages Architecture The most important concept in building Tonido applications is understanding how Tonido Service based architecture works. The closest analogy to Tonido services is Win32 message based APIs. 5.1. Services Services are runtime components that have well defined entry points. There are several services that already exist as part of the Tonido Core Runtime. Additionally, applications also contain one or more primary services. You interact with a service by sending a message. A message is a Tonido data construct that has specific structure. Here’s a high level example of how services operate:
A service when it starts up will register itself with the message dispatcher. We will talk more in detail about the message dispatcher, but for now assume that a message dispatcher is a registry of all services in the system and can route a given message to a service. Now let us say that a service (Service1) wants to interact with another service(Service2). It will create a message, address the message to the Service2 and send it to the Message Dispatcher. The Message Dispatcher takes the messages, checks the registry and invokes the Service::onMessage callback function in the appropriate service with the message as the payload. Tonido Platform Developers Guide © CodeLathe LLC
Page 27
The service will process the message, then optionally can create a response message and address it to service1 and post it back to the Message dispatcher. The Message Dispatcher will forward it to service1. When it is time to shutdown, the services will unregister themselves from the message dispatcher. This is the high level overview of Tonido services and message routing, now let us see how it works in practice. Tonido Service Interface A Tonido Service has the following interface that every service should implement. In C++ parlance, it is an abstract class whose interface MUST be implemented by any derived class. To write your own service, you will simply derive from this interface and implement the required APIs. class BOOTSTRAP_API Service { public: virtual
Service() {} ~Service() {}
//========================================================================== //! Initializes the Service /*! @param a_pluginPath is the path that the application loaded the plugin from @return true if the plugin was inited OK, false otherwise */ virtual bool init ( const std::string& a_pluginPath ) = 0; //========================================================================== //! Returns the name of the service that the dispatcher dispatches to /*! @return the name of the service */ virtual const std::string& name ( ) const = 0; //========================================================================== //! Starts the Service /*! Once the Service is started, the service must be ready to process incoming messages. @return false if there is an error, true otherwise */ virtual bool startup ( MessageDispatcher* dispatcher ) = 0; //========================================================================== //! Shutsdown the Service /*! The service must stop all activities once this occurs.
Tonido Platform Developers Guide © CodeLathe LLC
Page 28
@return true on success, false otherwise */ virtual bool shutdown() = 0; //@} //! @name Message Handler Entry //@{ //========================================================================== //! Message Handler Entry Point /*! */ virtual void onMessage(const void* pSender, Tonido::Data::Message::SharedPtr& args) = 0; //@} };
5.2. Messages Tonido Messages are the basic unit of communication within Services within a peer and between peers. A Message is the primary means by which services are invoked as well as how they communicate. A Message once created can be sent to another service either running in the same peer or in any other remote peer. A Tonido message consists of the following parts:
To Service From Service Operation To PeerID From PeerID AuthKey Variant Table Data Binary Data Payload
Tonido Platform Developers Guide © CodeLathe LLC
Page 29
To Service This is a string value of the service to which this message is intended. From Service This is a string value of the service from which the message originated. Operation This is a string value of the type of operation the message is trying to convey. To PeerID This is the Tonido ID of the Tonido instance to which the message is being sent From PeerID This is the Tonido ID of the Tonido instance from which the message is being sent AuthKey Stores authentication information related to the message VariantTable Data A message contains a flexible data storage mechanism called VariantTable. The advantage of storing information in the variant table is that the variant table data is automatically serialized for you when being transmitted across the network. Binary Data Payload Rest of the message contains an arbitrary data payload. You can store and send any binary data in that payload. Tonido will attempt to send huge message payloads directly across the network, but the transfer is not efficient, so it is recommended that you keep payloads to less than 500 KB for optimum network performance. 5.3. Variant Data C++ doesn’t have a native generic type. To help serialize messages with any type of data, one can use the Variant Data Types in Tonido. Variant Data types internally use boost:variant and can be serialized to binary and xml data. Variant Field A variant field stores a piece of data that can represent different types.
bool int double std::string int64_bt Tonido Platform Developers Guide © CodeLathe LLC
Page 30
uint64_bt uint32_bt RawData::SharedPtr Variant Row A variant row is a map of name, VariantField pairs.
Name 1 Name 2 .. Name n
VariantField 1 VariantField 2 .. VariantField n
Example of how to use a VariantRow VariantRow vr; vr["Alexander"] vr["Bool"] vr["Int"] vr["Double"] vr["UInt32"] vr["Int64"] vr["UInt64"]
= = = = = = =
"Great"; // std::string true; // bool 1; // int 5.5; // double (uint32_bt)45; // uint32_bt (int64_bt)1233456789; // int64 (uint64_bt)1233456789; // uint64
// … Check Data CHECK(vr["Alexander"].getString() == "Great"); CHECK(vr["Bool"].getBool() == true); CHECK(vr["Int"].getInt() == 1); CHECK(vr["Double"].getDouble() == 5.5); CHECK(vr["UInt32"].getUInt32() == 45); CHECK(vr["Int64"].getInt64() == 1233456789); CHECK(vr["UInt64"].getUInt64() == 1233456789); // … Write Variant Row into binary int vrSize = vr.getSize(); RawData::SharedPtr pRawData(new RawData(vrSize)); CHECK(vr.write(pRawData)); // … Read from binary VariantRow vrRead; CHECK(vrRead.read(pRawData)); Tonido Platform Developers Guide © CodeLathe LLC
Page 31
Variant Table A VariantTable is a map of name and Variant Row pairs.
Name 1 Name 2 .. Name n
VariantRow 1 VariantRow 2 .. VariantRow n
Example of how to use a VariantTable VariantTable vt; // … Row1 Data vt["Row1"]["Alexander"] = "Great"; // std::string vt["Row1"]["Bool"] = true; // bool vt["Row1"]["Int"] = 1; // int // … Row2 Data vt["Row2"]["Alexander"] = "Great"; // std::string vt["Row2"]["Bool"] = false; // bool vt["Row2"]["Int"] = 2; // int // … Check Data CHECK(vt["Row1"]["Alexander"].getString() == "Great"); CHECK(vt["Row1"]["Bool"].getBool() == true); CHECK(vt["Row1"]["Int"].getInt() == 1); CHECK(vt["Row2"]["Alexander"].getString() == "Great"); CHECK(vt["Row2"]["Bool"].getBool() == false); CHECK(vt["Row2"]["Int"].getInt() == 2); // … Write Variant Row into binary int vtableSize = vt.getSize(); RawData::SharedPtr pRawData(new RawData(vtableSize)); CHECK(vt.write(pRawData)); // … Read from binary VariantTable vtRead; CHECK(vtRead.read(pRawData));
Tonido Platform Developers Guide © CodeLathe LLC
Page 32
5.4. Dispatcher The MessageDispatcher object is a singleton object that is responsible for shuttling messages across services. It can do that, because all services in the system register themselves to the dispatcher who in turn invokes the service's onMessage member function when a message arrives for that service. The MessageDispatcher runs in its own thread. This means that when you send a message to a service, that service’s callback for your message will run in a different thread context. There are two message dispatchers in Tonido. 1. The primary dispatcher is the default dispatcher and handles all the routing. 2. The DeferredDispatcher. This message dispatcher is another dispatcher that can be used by services to defer longer running operations. When the message dispatcher invokes a function in a service, a service can do one of 3 things. 1. Handle the function in the Dispatcher thread context 2. Push the message to its own worker thread 3. Push the message to the system worker thread Note that handling the function in the Dispatcher thread context should only be done if the message handler work is minimal. Functions that take longer to execute should not run in the dispatcher context because they will prevent other messages from being processed in the entire system. 5.5. Common Service Operations Register Service Before your service can become an endpoint for communications, your service needs to register itself with the Message dispatcher. This should happen during Application “Startup” which we will talk about in more detail later. Service registration involves registering a service name string (e.g. “dummy”, and the associated function to invoke when messages arrive for that name string). See example below: MessageEventInfo::SharedPtr pmi(new MessageEventInfo()); pmi->m_event += Poco::Delegate(this, &DummyPlugin::onMessage); bool retval = m_pDispatcher->registerMessageHandler(this,pmi);
In the above code, to do a registration, you need to create a MessageEventInfo object and set the Delegate to the onMessage handler of your service (in this case the DummyPlugin Service) The name being registered is taken from the string returned by the service::name() function.
Tonido Platform Developers Guide © CodeLathe LLC
Page 33
onMessage Routine The onMessage routine is the main entry point for all messages being sent to your service. Depending upon the message operation value, your application will do various things here. UnRegister Service If a service is shutting down, then it MUST unregister itself from the message dispatcher. To unregister itself from the message dispatcher, it simply calls the unRegisterMessageHandler function of the MessageDispatcher. return m_pDispatcher->unRegisterMessageHandler(this);
Tonido Platform Developers Guide © CodeLathe LLC
Page 34
5.6. Deferred Dispatcher Behind the scenes, there is a separate Dispatcher object that runs on another thread from the main dispatcher. When a service registers with the dispatcher, the service is also registered silently with the deferred dispatcher. When a service unregisters from the dispatcher, the service is also unregistered from the deferred dispatcher. When a service gets a message, the onMessage function is invoked at the main dispatcher thread context. Since the main dispatcher thread services messages for the whole of Tonido, if the application takes a long time to respond, it will starve message processing for other apps. To avoid this problem, the application service can post the message to the deferred dispatcher who will call the service back in the deferred dispatcher thread context. This allows the service to do somewhat longer running operations without blocking the entire Tonido application message queue. Note that the deferred dispatcher should not be used for really long operations. If you need to run longer running operations, plan to start your own thread.
Tonido Platform Developers Guide © CodeLathe LLC
Page 35
6. Application Basics 6.1. Tonido Directory Layout Tonido has specific conventions on how everything is laid out in the Tonido Install folder. Install Directory Layout The following is the Tonido Install directory layout. Tonido installs to the following locations by default. Windows %APPDATA%\tonido Linux ~/tonido Mac OSX ~/tonido
Tonido Install Folder + manifest.xml + config.xml + localconfig.xml + Tonido Executable + bootstrap.dll (.so) + core.dll (.so) + thirdparty files +-------------Plugins +--------search + manifest.xml + icon.gif + iconbg.gif + search.dll (.so) + search.zip +------thots + manifest.xml + icon.gif + iconbg.gif + thots.dll (.so) + thots.zip
manifest.xml A manifest file contains meta information that Tonido uses to load an application as well as for updating a Tonido component automatically. We will describe the format of a Tonido component in depth in another section.
Tonido Platform Developers Guide © CodeLathe LLC
Page 36
config.xml A config xml file contains configurable URLs that Tonido uses for various actions. It includes the URL for Tonido news, the URL for list of new applications, the URL for Tonido domain servers and so on. localconfig.xml A localconfig xml file contains the location of the Tonido data directory. The default location is specified by using a special value HOME. If any other value is specified other than HOME, then the location is interpreted as a file path. A relative path can also be specified if you want Tonido to be fully self contained for example in a USB key install. bootstrap.dll(so) This is part of the Tonido Core runtime. It contains APIs, many services, systems for loading, unloading applications etc. The code of the bootstrap component is open and available as part of the Tonido SDK. core.dll(so) This contains many of the Tonido core services including Tonido P2P Network services, the HTTP Services etc. Thirdparty files Other required thirdparty library files like Poco dlls etc are distributed as part of the Tonido install.
Tonido Platform Developers Guide © CodeLathe LLC
Page 37
Tonido Data Directory Layout Tonido Data Directory + configex.xml + domainlist.xml + ini.db +-------------logs tonido.log +--------errorlogs +-------------profiles +--------profile1 + profile.png + profile.db + profile.vr + ------------plugindata + ------------plugin1 + ------------plugin2 + ------------userdata + ------------plugin1 + ------------plugin2 +------profile2
configex.xml This xml file controls many of the configurable behaviours of Tonido. domainlist.xml This xml file contains the list of Tonido domain servers that are used when creating a profile. The actual server used is randomly chosen in the list. This allows some kind of distribution of load balancing among different TDS. For now, there is only one Tonido DS (at tonidoid.com) ini.db This is an unencrypted sqlite database that contains several global configuration values. tonido.log This is the runtime log for Tonido. Useful for debugging problems and figuring out other problems. Errorlogs directory Under some conditions, Tonido will dump a binary dump called errorlog into the error logs directory. Only the last n error logs are stored to prevent taking up too much disk space.
profiles directory This contains the list of profiles that are available locally. The name of the profile is saved as the name of the directory
Tonido Platform Developers Guide © CodeLathe LLC
Page 38
profile.png This is the image of the Tonido profile profile.vr This is a XML file that contains more profile specific information. profile.db This is an encrypted sqlite database that is encrypted by a SHA-1 hash of the user password. The database contains all the Core data, like profile information, Tonido messages, Tonido Groups data, Activity data and so on. plugindata directory This directory has sub folders for each application. Each application will store application specific data inside this directory. userdata directory This directory has sub folders for each application. Each application will store user viewable data inside this directory. For example, received photos via the Tonido Photos application will be stored in this directory. 6.2. Application Lifecycle
Init
Load
Startup
Unloaded
Running
Shutdown
Load Application Initially, when Tonido starts, it will enumerate all applications listed in the plugins directory. If the application is not suspended (there are no .suspend files), it will load the application shared library into Tonido process memory. Tonido Platform Developers Guide © CodeLathe LLC
Page 39
Init Service Each service in the application is inited. The service in the application will use this time to setup required stuff. The init function is called before the plugin service is started and allows the plugin service to do any required initialization. Note that the plugin is called with the base path from where the plugin is being loaded from. The base path is useful if the plugin wants to locate other files that are shipped along with itself. Startup Service Each service in the application is started. The service in the application will use this time to register itself with the dispatcher, send any other messages to other services. The startup is called after init and the plugin is responsible to ready itself for receiving messages via the onMessageHandler right after this call. Every plugin SHOULD do these during startup: a) Register the service with the MessageDispatcher Optionally, every plugin can do these other things a) b) c) d)
Initialize the statgroup Start any threads Register any recurring timers Register for event notifications
Running (Handle onMessage) The application during this time is running, it can handle any onMessage calls called by the Message Dispatcher. This is the main service request handler that every plugin should implement. Any message being routed to this function indicates an incoming message from a service originating locally or remotely that wants to send a message to this specific name. Shutdown Service When Tonido wants to shutdown an application, it will invoke Shutdown, the service needs to unregister itself from the message dispatcher and get ready to exit. The shutdown function is called when the plugin is being shutdown. This could be due to several events, a) The user suspending the plugin b) The user uninstalling the plugin c) The user logging out of a profile which causes all plugins to be unloaded
Tonido Platform Developers Guide © CodeLathe LLC
Page 40
The plugin SHOULD do the following things shutdown: a) b) c) d)
Unregister itself from the message dispatcher Stop any threads Do any other cleanup required (say close database handles etc) Unregister from event notifications and timers
Unload Application Finally the shared library of the application is unloaded from Tonido’s memory. 6.3. Manifests Manifests store meta data about applications. Tonido checks the manifest before loading applications. Tonido also uses manifests to update applications when an update is available. tonido http://patch.codelathe.com/tonido/live/update/core/x86win32/ http://www.tonido.com 0.2.0.7727 \d+\.\d+\.\d+\.\d+ Windows NT IA32 CodeLathe Tonido Core Package false BugTrapU.dll 1 738E64A658F29DBE6453A825130859B389DC0F1DAAB20B3438736298396FCE5E18A23A5C812E914F4BD 42251C1C0BD1420621B99FF1C
A manifest is an xml file with four main sections. Meta This contains most of the meta data related to the application. Name
Tonido Platform Developers Guide © CodeLathe LLC
Should be same as the application directory name. It is recommended to keep everything lower case to avoid any issues Page 41
BaseURL
HomePageURL Version Compatible Version
OS OSVersion
Arch
Vendor Description Critical
This is the HTTP URL where the update information can be found if any. If empty, the application check for update is skipped. Note that the BaseURL+manifest.xml file MUST exist. The website for the application Current Plugin version This is a regular expression which describes the compatibility of the application with the version of the Tonido Core Runtime. Describes which OS the application will run on. Describes which specific OS version the application will run on. If empty this means this is not considered during loading. This is the CPU architecture with which the application is compatible with. The author of this application A short description of the application {true|false} if true, this application update will show up on the user’s update available notification as a critical update available.
Files This contains the list of files in that application. Every file element also has other XML attributes. Checksum
Path Executable
Tonido Platform Developers Guide © CodeLathe LLC
MD5 Checksum of the file. Tonido uses the checksum to first check if the local file is already up to date. In that case it will avoid doing another update. It also uses the checksum to verify that the download of the file via HTTP worked correctly without any corruption. The location of the file {0|1} 1 means Tonido on doing an update will set the executable bit on Page 42
the file
Shared Libraries If an application depends upon a shared library, it can list it in the Shared Libraries section and Tonido will explicitly load that shared library before it loads the application. It will also unload the shared library in case the application gets unloaded. Signature Data Because downloading files can be dangerous and to prevent attacks that circumvent the Tonido patching process, Tonido requires that any manifests that requires files to be downloaded to be signed. Currently CodeLathe owns the private key and the public key is in every Tonido runtime. This prevents someone creating a dummy manifest and making Tonido download that manifest and downloading unknown files into the user’s machine.
Tonido Platform Developers Guide © CodeLathe LLC
Page 43
7. HTTP Interface Tonido includes an embedded HTTP server in the runtime. This server handles incoming HTTP calls, transforms it into Tonido Messages and sends it to the appropriate service.
Incoming HTTP Request
Translate Request to Message
Request SyncProxy Service to send Message to Dispatcher
Dispatcher invokes service for given message
Service responds back to SyncProxy Service with the HTTP response
Response sent back to browser
Tonido Platform Developers Guide © CodeLathe LLC
Page 44
A request arrives from the web browser via HTTP GET or POST. This is intercepted by the embedded HTTP Web Server that takes the request and parses it into a Message. It then posts this message to the Message dispatcher and waits for the response to come back. Once the response is received it sends it back to the browser. The web server expects URL to be of a certain form:
http://host:port/app/SERVICE/OPERATION {? OPTIONAL GET Params } See the following example: GET http://somehost/app/photos/homepage?user=id
The app tells the web browser that this is an application level request that needs the URLs to be handled specifically. The URL token following application is considered the Service to which the URL is being sent to. In the example, this will be sent to the photos Service. The rest of the URL is considered as the operation. In this case (homepage) Note that GET parameters, (user=id) will be sent to the service as well as part of the message. If the request is a POST, the body of the POST will be sent in the message to the service. 7.1. Anatomy of a HTTP Request This section assumes that you have a basic working knowledge of HTTP. To Service: SERVICE Operation: OPERATION From Service: SyncProxy Service AuthKey: AuthSource UI AuthKey: AuthLocation Varies AuthKey: AuthRole Owner or Anonymous
VARIANT TABLE DATA Defs::HTTP::MS_HTTP_REQ
Defs::HTTP::S_HTTP_REQ_REQUEST
std::string
URL Query (The part after ? in a URL)
Defs::HTTP::MS_HTTP_REQ Defs::HTTP::MS_HTTP_REQ
Defs::HTTP::S_HTTP_REQ_METHOD Defs::HTTP::S_HTTP_REQ_URL
std::string std::string
“GET” or “POST” Full URL except HOST and QUERY /app/service/operation
Defs::HTTP::MS_HTTP_REQ Defs::HTTP::MS_HTTP_REQ
Defs::HTTP::S_HTTP_REQ_CLIENT_IP Defs::HTTP::S_HTTP_REQ_CLIENT_PO RT
std::string uint32_bt
IP address of the client (e.g. 127.0.0.1) Port of the client (e.g. 2468)
Tonido Platform Developers Guide © CodeLathe LLC
Page 45
Defs::HTTP::MS_HTTP_REQ
Defs::HTTP::S_HTTP_REQ_HOST_IP
std::string
Defs::HTTP::MS_HTTP_REQ
Defs::HTTP::S_HTTP_REQ_HOST_PORT
uint32_bt
Defs::HTTP::MS_HTTP_REQ Defs::HTTP::MS_HTTP_REQ
Defs::HTTP::S_HTTP_REQ_SERVER_IP Defs::HTTP::S_HTTP_REQ_SERVER_PO RT Defs::HTTP::S_HTTP_REQ_CREDENTIA LS_SCHEME Defs::HTTP::S_HTTP_REQ_CREDENTIA LS_INFO Defs::Sync::S_SYNC_TOKEN
std::string uint32_bt
IP Address of Host as found in the Host Header Field 127.0.0.1 Port Number of Host found in the Host Header Field 10001 IP Address of Server 127.0.0.1 Port of the server 10001
std::string
Basic
uint32_bt
HTTP Basic Auth Data “username:password” UUID Token for this request
Defs::HTTP::MS_HTTP_REQ Defs::HTTP::MS_HTTP_REQ Defs::Sync::MS_SYNC
std::string
BINARY DATA PAYLOAD HTTP POST Data
7.2. Anatomy of a HTTP Response To respond back to a HTTP request, one can set the following in a message To Service: SyncProxyService Operation: MO_SYNC From Service: Service
VARIANT TABLE DATA Defs::HTTP::MS_HTTP_RESP
Defs::HTTP::S_HTTP_RESP_STATUS
int
HTTP Status return
Defs::HTTP::MS_HTTP_RESP Defs::Sync::MS_SYNC Defs::HTTP::MS_HTTP_RESP Defs::HTTP::MS_HTTP_RESP Defs::Sync::MS_SYNC
Defs::HTTP::S_HTTP_RESP_TYPE Defs::Sync::S_SYNC_TOKEN Defs::HTTP::S_HTTP_CONTENT_DISPOSITION Defs::HTTP::S_HTTP_RESP_TRANSFER_ENCODING Defs::Sync::S_SYNC_PIECE_NUM
std::string std::string std::string std::string int
“text/html” Same ID as the HTTP request
-1 marks completion
BINARY DATA PAYLOAD HTTP Response Data
Tonido Platform Developers Guide © CodeLathe LLC
Page 46
7.3. HTTP Core Authentication Most of the Tonido HTTP APIs required that a user is authenticated first. Tonido APIs use cookies to verify that a user has been authenticated. Here’s the detailed flow of how an application can authenticate itself to the Tonido Core Runtime.
Here’s the transcript of the login process CLIENT HTTP GET /core/getauthenticationinfo SERVER { PROFILE NAME } { USER DISPLAY NAME } id> { USER ID } { 0 | 1 } { 0 | 1 } 0 CLIENT HTTP POST /core/loginprofile profile=mike&password=4398439849234&safemode=0&autologin=1 SERVER Set Cookie and send XML data loginprofile Tonido Platform Developers Guide © CodeLathe LLC
Page 47
{ 0 | 1 | 2 } { if Remote Login (2) Secret Qn } -------------------OPTIONAL (if logging in remotely) CLIENT HTTP POST /core/authenticateprofile secretqn=question&secretans=ans&password=29832j2kj32j32 SERVER Set Cookie and send XML data loginprofile { 0 | 1 } --------------------
8. P2P Interface Tonido allows applications to connect and communicate with other Tonido instances directly via P2P.
Tonido Platform Developers Guide © CodeLathe LLC
Page 48
Application creates Message to send over Network to remote peer
Application creates wrapper message, serializes the message into the wrapper
Application sends wrapper message to Network Service
Network Service locates remote Peer and transmits the message to the peer
Message Transmitted across network to remote peer
Remote Peer receives message, deserializes the message and forwards it to actual service endpoint
Tonido Platform Developers Guide © CodeLathe LLC
Page 49
Network Message Payloads To send a message over the network, you need to create a message and serialize it into a binary payload and stuff it into the body of another message that you send to the Network Service.
Network Message To Service: NetworkService Operation:
Actual Message: Serialized into Network Message Body
SEND
ToService: RemoteService Operation: Remote Operation To Peer: Remote Peer AUTHKEY
Contents
From Peer: Our Peer
Process Walkthrough Step 1: Create a new Message to store your payload. Use the utility function to set required parameters. Store your data in n the message. static Tonido::Data::Message::SharedPtr createNewNetworkMessage ( const std::string& a_toService, const std::string& a_operation, const Tonido::Data::PeerID& a_toPeer, const std::string& a_fromService );
Tonido Platform Developers Guide © CodeLathe LLC
Page 50
Step 2: Create a new Network message using the original message and the Authorization Key and Authorization Context static Tonido::Data::Message::SharedPtr prepareNetworkMessage ( Tonido::Data::Message::SharedPtr& a_pMessage, Tonido::Data::AuthKey::AuthRole a_role, const std::string& a_authID = "" );
Step 3: Send the network message to the service Tonido::Dispatcher()->postMessage(pNetworkMessage);
Tonido Platform Developers Guide © CodeLathe LLC
Page 51
9. Authentication and Security 9.1. Tonido Security Tonido’s implementation goes to great lengths to preserve security. There are two ways external data can enter Tonido: 1. Via HTTP 2. Via Tonido Network As a result, Tonido looks at every message that enters the system and tags it with an AuthKey on the security level associated with that input. This allows downstream services which get that input to act accordingly. This is not a bullet proof system and it puts the onus on the application to ensure the security of all operations and it requires them to understand what they are doing. Additionally, here are some additional ways Tonido is secure: • • • • •
All access is turned off by default. Public HTTP access is disabled for example. Tonido runs as low-privilege process in Windows even when run by an administrator Tonido does not allow profile creation and deletion from remote locations by default Tonido underwent security and penetration audit by a security services firm and all known issues were addressed Remote login into Tonido profile requires two level authentication
9.2. AuthKeys Every message in Tonido has an Authkey associated with it. The AuthKey information is filled out before the message is forwarded to a local service. A service must always inspect the AuthKey of a message before performing the operation.
Tonido Platform Developers Guide © CodeLathe LLC
Page 52
AuthSource Determines from where the message originated. • • • •
Internal : Message originated locally from a service UI: Message originated from the UI Network: Message Originated from Network Unknown: Uninitialized
Unknown
Network
Auth Source
Internal
UI (HTTP)
Tonido Platform Developers Guide © CodeLathe LLC
Page 53
AuthLocation Determines the physical location from where the message originated. • • • •
Local: Message originated locally 127.0.0.1 Trusted: Message originated from trusted locations Remote Access: Message originated from Tonido Network based Remote Access Remote: Message originated remotely
Remote
Remote Access
Auth Location
Local
Trusted
Tonido Platform Developers Guide © CodeLathe LLC
Page 54
AuthRole Determines who is sending this message. • • • •
Owner: Message originated with Owner credentials Group Admin: Message originated with Group Admin credentials Group: Message originated with Group credentials Anonymous: Message originated with Anonymous credentials
Anonymous
Group
Auth Role
Owner
Group Admin
Tonido Platform Developers Guide © CodeLathe LLC
Page 55
10.
Static APIs
Tonido provides several static APIs that can be called directly by applications. In general, Applications should use the APIs available in the Tonido::API namespace. These are wrapper functions over class member functions. See TonidoAPI.h for details on these functions. 10.1. Profile Manager APIs Manages Tonido User Profiles 10.2. Email Manager APIs Sends Emails via the Tonido Domain Server 10.3. Activity Manager APIs Manages Tonido Activities that appear in the Tonido Admin UI 10.4. Peer Manager APIs Manages information related to other Tonido instances 10.5. Group Manager APIs Manages Tonido Groups and actions related to those groups 10.6. Mail Manager APIs Sends Tonido Messages from one Tonido instance to another 10.7. Network Manager APIs Manages Tonido Network connections 10.8. Remote Access Manager APIs Manages Remote Access Information 10.9. Image Manipulation Provides APIs for image manipulation including resizing etc 10.10. Virtual FS Allows you to read and load files from zip files.
Tonido Platform Developers Guide © CodeLathe LLC
Page 56
11.
Service APIs
11.1. Timer This is the master timer service for Tonido. • • • • •
Any service wishing to receive a timer message at a fixed interval can register with the TimerService by sending it a "Start" message. From that point on, they will get messages with a "Timer" operation every fixed interval. If they don't wish to get the timer message, they can send a "Stop" message and they will be removed from future timer messages. Currently the timer resolution is 1000 msec This is useful if the service wants a timer to process data at specific intervals but doesnt want to setup individual timers which can be expensive and also can have thread synchronization implications for the callbacks.
11.2. Event Notifier Service EventNotifier Service helps multiple services be notified when an event occurs. • • •
A Service wishing to register for a particular event can “Subscribe” for that event A service can signal that event by sending a “Signal” message The “Signal” Event is propagated to those who registered
Tonido uses EventNotifier extensively to communicate various system events to interested services. These events include • • • • • • • • • • • • • • • • •
Profile Login Profile Logout App Startup App Shutdown Network Enable Network Disable Network Online Network Offline Peer Connection Established Peer Disconnect Peer Online Peer Offline Group Added Group Removed User Added to Group User Removed from Group Group Invitation Accepted
Tonido Platform Developers Guide © CodeLathe LLC
Page 57
11.3. FileTransfer File Transfer can transfer any file to another Tonido peer. To initiate a transfer, simply send a FileStart message to the service. Once the file transfer is complete, a file complete message is sent back to the originating service. 11.4. Network The Tonido Network service is responsible for transmitting messages to Tonido Peers. 11.5. MachineInfo This service periodically captures machine information like CPU load, disk size etc. 11.6. Sync Proxy This service allows you to send a message and block till a reply to that message is returned. For example, this is how the HTTP Service sends incoming HTTP requests into the Tonido system. 11.7. Stats Service Manages statistics from various services and consolidates them. A service wishing to generate stats needs to instantiate one of the StatsGroup objects . Statistics will be periodically sent out to the Stats Service which will consolidate them for potential display in the Tonido Admin.
Tonido Platform Developers Guide © CodeLathe LLC
Page 58
Tonido Platform Developers Guide © CodeLathe LLC
Page 59
12.
Guide to Building User Interfaces for Tonido Applications
Coming Soon
13.
Third party Libraries
Tonido SDK uses several third party libraries. 13.1. POCO POCO or Portable Components is a high-quality, cross platform C++ library. Tonido uses many code components from POCO. Since Tonido links with POCO already, a Tonido application can use POCO quite easily. There are no extra libraries to ship. See http://www.pocoproject.org/ 13.2. Boost Boost provides free peer-reviewed high quality portable C++ libraries. Tonido uses Boost Shared Pointers. See http://www.boost.org/ 13.3. GD GD is an open source library meant for dynamic creation of images. GD can work with PNG, JPEG and GIF images. The Tonido::Data::Image class is a wrapper around GD and allows you to resize images etc. See http://www.libgd.org/ 13.4. Sqlite Sqlite implements a self-contained, serverless, zero-config, transactional SQL database engine. See http://www.sqlite.org/ 13.5. Crypto++ Crypto++ is a free C++ class library of cryptographic schemes including hash functions, AES, publickey cryptography, key agreement schemes and more. See http://www.cryptopp.com/ 13.6. OpenSSL OpenSSL is a robust toolkit that implements the Secure Sockets Layer and Transport Layer Security protocols as well as a full strength general purpose cryptography library. See http://www.openssl.org/ 13.7. Lua Lua is a powerful, fast, lightweight, embeddable scripting language. See http://www.lua.org/ 13.8. LuaBind LuaBind is a library that helps you create bindings between C++ and Lua. See http://www.rasterbar.com/products/luabind.html
Tonido Platform Developers Guide © CodeLathe LLC
Page 60
14.
Patching System
Coming soon
Tonido Platform Developers Guide © CodeLathe LLC
Page 61
15.
Listing your Plugin in Tonido App Store
The Tonido App store will function as a repository for users to easily download and install new applications. We envision both free and commercial applications being available through the store. Developers who wish to list their applications can submit it to the store and it will be added to the listing. Developers will also be able to submit commercial applications, set a price and have proceeds of the sales be sent to them automatically. More details on this coming soon.
Tonido Platform Developers Guide © CodeLathe LLC
Page 62
16. • • • • •
Recommendations, Tips and Tricks Applications have to put security first and make sure all incoming onMessage calls have correct security processing. Applications have to be good citizens in the Tonido eco-system and in the user’s computer. o This means they have to mindful of CPU, Hard disk, memory resources Applications need to remember that they are sharing the same process space as the rest of the Tonido, so they need to be careful not to introduce instability All exceptions need to be caught and handled by the application. An uncaught exception will currently cause Tonido to shutdown and exit Applications need to use Shared Pointers whenever possible to avoid memory leaks.
16.1. Running multiple Tonido instances on a single machine During development it is required to run many instances of Tonido in a single machine, especially when testing P2P interactions. It is pretty simple to set this up. • • •
•
•
Open a command window Go to the debug directory where tonidoconsole is located Start Tonido, using the following command line tonidoconsole o E.g. tonidoconsole instance1 10001 will start Tonido using instance1 as its datadirectory and 10001 as the HTTP port Start any number of new Tonido instances using the same command line syntax in different command windows o E.g. tonidoconsole instance2 10002 and so forth. Ensure that no two instances share the same data directory and the same HTTP port
Tonido Platform Developers Guide © CodeLathe LLC
Page 63
