In Part 1 of this post we talked about how Force.com platform manages the Data model. In this post we’ll describe about Multitenant App Development.. Read on…
Multitenant App Development
The previous section explains the architecture that Force.com uses to store metadata and data. This section briefly explains how app developers can create a schema’s underlying metadata and then build apps that manage data.
THE FORCE.COM BROWSER-BASED DEVELOPMENT ENVIRONMENT
Developers can declaratively build server-side application components using the Force.com Web browser-based development environment, commonly referred to as the Force.com Setup screens. This point-and-click UI supports all facets of the application schema building process, including the creation of an application’s data model (objects and their fields, relationships, etc.), security and sharing model (users, profiles, role hierarchies, etc.), user interface (screen layouts, data entry forms, reports, etc.), declarative logic (workflows), and programmatic logic (stored procedures and triggers). For example, the following screen is the Force.com Schema Builder, an intuitive, ERD-like data modeling tool.
Force.com Setup screens provide access to many built-in system features that make it easy to implement common application functionality without writing the otherwise complicated and error-prone code that’s required in traditional database systems.
Some such features include the following.
- Force.com native user interfaces are easy to build because there’s no coding necessary. Behind the scenes, a native app UI supports all the usual data access operations, including queries, inserts, updates, and deletes. Each data manipulation operation performed by native platform applications can modify one object at a time and automatically commit each change in a separate transaction.
- A workflow is a declarative trigger—a predefined action triggered by the insert or update of a row. A workflow can update to a field or send an outbound message. Workflow rules specify the criteria that determine when to trigger a workflow action. For example, you might declare a workflow that, immediately after a record is updated, automatically updates the row’s Status field to “Modified.” All workflow operations occur within the context of the transaction that triggers the workflow. If the system rolls back a transaction, all related workflow operations that were executed also roll back.
- When defining a text field for an object that contains sensitive data, developers can easily configure the field so that Force.com encrypts the corresponding data and, optionally, uses an input mask to hide screen information from prying eyes. Force.com encrypts fields using AES (Advanced Encryption Standard) algorithm 128-bit keys.
- A declarative validation rule is a simple way for an organization to enforce a domain integrity rule without any programming. For example, you might declare a validation rule that makes sure that a LineItem object’s Quantity field is always greater than zero.
- A formula field is a declarative feature of Force.com that makes it easy to add a calculated field to an object. For example, you might add a field to the LineItem object to calculate a LineTotal value.
- A roll-up summary field is a cross-object field that makes it easy to aggregate child field information in a parent object. For example, you might create an OrderTotal summary field in the SalesOrder object based on the LineTotal field of the LineItem object.
Note: Internally, Force.com implements formula and roll-up summary fields using native database features and efficiently recalculates values synchronously as part of ongoing transactions.
APIS
Force.com provides open, standards-based APIs that developers can use to build apps. Both RESTful and Web services (SOAP-based) APIs are available that provide access to Force.com’s many features. Using these various APIs, an application can do many things such as:
- Manipulate metadata that describes an application schema
- Create, read, update, and delete (CRUD) business data
- Bulk-load a large number of records asynchronously
- Expose a near real-time stream of data in a secure and scalable way
- Embed social networking functionality into an application with minimal effort
- Add social collaboration features to any application, including push notifications and data feeds
For more information about each API, see the Integration section of Developer Force.
QUERY LANGUAGES
Apps can use the Salesforce Object Query Language (SOQL) to construct simple but powerful database queries. Similar to the SELECT command in the Structured Query Language (SQL), SOQL allows you to specify the source object, a list of fields to retrieve, and conditions for selecting rows in the source object. For example, the following SOQL query returns the value of the Id and Name field for all Account records with a Name equal to the string ‘Acme’.
SELECT Id, Name FROM Account WHERE Name = 'Acme'Force.com also includes a full-text, multi-lingual search engine that automatically indexes all text-related fields. Apps can leverage this pre-integrated search engine using the Salesforce Object Search Language (SOSL) to perform text searches. Unlike SOQL, which can only query one object at a time, SOSL enables you to search text, email, and phone fields for multiple objects simultaneously. For example, the following SOSL statement searches for records in the Lead and Contact objects that contain the string ‘Joe Smith’ in the name field and returns the name and phone number field from each record found.
FIND {"Joe Smith"} IN Name Fields RETURNING lead(name, phone), contact(name, phone)APEX AND PROCEDURAL LOGIC
Apex, which is similar to Java in many respects, is a powerful development language that developers can use to centralize procedural logic in their application schema. Apex code can declare program variables and constants, execute traditional flow control statements (if-else, loops, etc.), perform data manipulation operations (insert, update, upsert, delete), and transaction control operations (setSavepoint, rollback).
You can store Apex programs in Force.com using two different forms: as a named Apex class with methods (akin to stored procedures in traditional database parlance) that applications execute when necessary, or as a database trigger that automatically executes before or after a specific database manipulation event occurs. In either form, Force.com compiles Apex code and stores it as metadata in the UDD. The first time that an organization executes an Apex program, Force.com’s runtime interpreter loads the compiled version of the program into an MRU cache for that organization. Thereafter, when any user from the same organization needs to use the same routine, Force.com can save memory and avoid the overhead of recompiling the program again by sharing the ready-to-run program that is already in memory.
With the addition of a simple keyword here and there, developers can use Apex to support many unique application requirements. For example, developers can expose a method as a custom RESTful or SOAP-based API call, make it asynchronously schedulable, or configure it to process a large operation in batches.
Apex is much more than “just another procedural language.” It’s an integral Force.com component that helps the system deliver reliable multitenancy. For example, Force.com automatically validates all embedded SOQL and SOSL statements within a Force.com class to prevent code that would otherwise fail at runtime. Force.com then maintains corresponding object dependency information for valid Force.com classes and uses it to prevent changes to metadata that would otherwise break dependent code.
Many standard Apex classes and system static methods provide simple interfaces to underlying system features. For example, the system static DML methods such as insert, update, and delete have a simple Boolean parameter that developers can use to indicate the desired bulk processing option (all or nothing, or partial save); these methods also return a result object that the calling routine can read to determine which records were unsuccessfully processed and why. Other examples of the direct ties between Apex and Force.com features include the built-in email classes and XmlStream classes, just to name a couple.
Multitenant Processing
In large part, Force.com performs and scales so well because the engineers at salesforce.com built the service with two important principles in mind.
- Do everything as efficiently as possible.
- Help developers do everything as efficiently as possible.
With these thoughts in mind, the following sections explain some of the unique processing architectures of Force.com.
MULTITENANT QUERY PROCESSING
Most modern database systems determine optimal query execution plans by employing a cost-based query optimizer that considers relevant statistics about target table and index data. However, conventional, cost-based optimizer statistics are designed for single-tenant applications and fail to account for the data access characteristics of any given user executing a query in a multitenant environment. For example, a given query that targets an object with a large volume of data would most likely execute more efficiently using different execution plans for users with high visibility (a manager that can see all rows) versus users with low visibility (sales people that can only see rows related to themselves).
To provide sufficient statistics for determining optimal query execution plans in a multitenant system, Force.com maintains a complete set of optimizer statistics (tenant-, group-, and user-level) for each virtual multitenant object. Statistics reflect the number of rows that a particular query can potentially access, carefully considering overall tenant-specific object statistics (e.g., the total number of rows owned by the tenant as a whole), as well as more granular statistics (e.g., the number of rows that a specific privilege group or end user can potentially access).
Force.com also maintains other types of statistics that prove helpful with particular queries. For example, the service maintains statistics for all custom indexes to reveal the total number of non-null and unique values in the corresponding field, and histograms for picklist fields that reveal the cardinality of each list value.
When existing statistics are not in place or are not considered helpful, Force.com’s optimizer has a few different strategies it uses to help build reasonably optimal queries. For example, when a query filters on the Name field of an object, the optimizer can use the MT_Fallback_Indexes table to efficiently find requested rows. In other scenarios, the optimizer will dynamically generate missing statistics at runtime.
Used in tandem with optimizer statistics, Force.com’s optimizer also relies on internal security-related tables (Groups, Members, GroupBlowout, and CustomShare) that maintain information about the security domains of system users, including a given user’s group memberships and custom access rights for object and rows. Such information is invaluable at determining the selectivity of query filters on a per-user basis. See the paper “Understanding Force.com system and data access controls” for more information about the platform’s embedded security model.
The flow diagram in the following figure illustrates what happens when Force.com intercepts a request for data that is in one of the large heap tables such as MT_Data. The request might originate from any number of sources, such as an API call or stored procedure. First, Force.com executes “pre-queries” that consider the multitenant-aware statistics. Then, considering the results returned by the pre-queries, the service builds an optimal underlying database query for execution in the specific setting.
As the following table shows, Force.com can execute the same query four different ways, depending on who submits the query and the selectivity of the query’s filter conditions.
| … nested loops join, drive using view of rows that user can see. | ||
| … use of index related to filter. | ||
| … ordered hash join, drive using MT_DATA. | ||
| … use of index related to filter. | ||
MULTITENANT SEARCH PROCESSING
Web-based application users have come to expect an interactive search capability to scan all or a selected scope of an application’s database, return up-to-date ranked results, and do it all with sub-second response times. To provide such robust search functionality for applications, Force.com uses a search engine that is separate from its transaction engine. The relationship between the two engines is depicted in the following figure. The search engine receives data from the transactional engine, with which it creates search indexes. The transactional engine forwards search requests to the search engine, which returns results that the transaction engine uses to locate rows that satisfy the search request.
As applications update data in text fields (CLOBs, Name, etc.), a pool of background processes called indexing servers are responsible for asynchronously updating corresponding indexes, which the search engine maintains outside the core transaction engine. To optimize the indexing process, Force.com synchronously copies modified chunks of text data to an internal “to-be-indexed” table as transactions commit, thus providing a relatively small data source that minimizes the amount of data that indexing servers must read from disk. The search engine automatically maintains separate indexes for each organization (tenant).
Depending on the current load and utilization of indexing servers, text index updates may lag behind actual transactions. To avoid unexpected search results originating from stale indexes, Force.com also maintains an MRU (most recently used) cache of recently updated rows that the system considers when materializing full-text search results. Force.com maintains MRU caches on a per-user and per-organization basis to efficiently support possible search scopes.
Force.com’s search engine optimizes the ranking of records within search results using several methods. For example, the system considers the security domain of the user performing a search and puts more weight in the rows that the current user can access. The system can also consider the modification history of a particular row and rank more actively updated rows ahead of those that are relatively static. The user can choose to weight search results as desired, for example, placing more emphasis on recently modified rows.
MULTITENANT BULK OPERATIONS
Transaction-intensive applications generate less overhead and perform much better when they combine and execute repetitive operations in bulk. For example, contrast two ways an application might load many new rows. An inefficient approach would be to use a routine with a loop that inserts individual rows, making one API call after another for each row insert. A much more efficient approach would be to create an array of rows and have the routine insert all of them with a single API call.
Efficient bulk processing with Force.com is simple for developers because it is baked into API calls. Internally, Force.com also bulk processes all internal steps related to an explicit bulk operation.
Force.com’s bulk processing engine automatically accounts for isolated faults encountered during any step along the way. When a bulk operation starts in partial save mode, the engine identifies a known start state and then attempts to execute each step in the process (bulk validate field data, bulk fire pre-triggers, bulk save records, etc.). If the engine detects errors during any step, the engine rolls back offending operations and all side effects, removes the rows that are responsible for the faults, and continues, attempting to bulk process the remaining subset of rows. This process iterates through each successive stage until the engine can commit a subset of rows without any errors. The application can examine a return object to identify which rows failed and what exceptions they raised.
Note: At your discretion, an all-or-nothing mode is available for bulk operations. Also, the execution of triggers during a bulk operation is subject to internal governors that restrict the amount of work.
MULTITENANT SCHEMA MODIFICATIONS
Certain types of modifications to the definition of an object require more than simple UDD metadata updates. In such cases, Force.com uses efficient mechanisms that help reduce the overall performance impact on the cloud database service at large.
For example, consider what happens behind the scenes when you modify a column’s datatype from picklist to text. Force.com first allocates a new slot for the column’s data, bulk-copies the picklist labels associated with current values, and then updates the column’s metadata so that it points to the new slot. While all of this happens, access to data is normal and applications continue to function without any noticeable impact.
As another example, consider what happens when you add a roll-up summary field to an object. In this case, Force.com asynchronously calculates initial summaries in the background using an efficient bulk operation. While the background calculation is happening, users who view the new field receive an indication that Force.com is currently calculating the field’s value.
MULTITENANT ISOLATION AND PROTECTION
To prevent malicious or unintentional monopolization of shared, multitenant system resources, Force.com has an extensive set of governors and resource limits associated with Force.com code execution. For example, Force.com closely monitors the execution of a code script and limits how much CPU time it can use, how much memory it can consume, how many queries and DML statements it can execute, how many math calculations it can perform, how many outbound Web services calls it can make, and much more. Individual queries that Force.com’s optimizer regards as too expensive to execute throw a runtime exception to the caller. Although such limits might sound somewhat restrictive, they are necessary to protect the overall scalability and performance of the shared database system for all concerned applications. In the long term, these measures help to promote better coding techniques among developers and create a better experience for everyone that uses the platform. For example, a developer who initially tries to code a loop that inefficiently updates a thousand rows one row at a time will receive runtime exceptions due to resource limits and then begin using Force.com’s efficient bulk processing API calls.
To further avoid potential system problems introduced by poorly written applications, the deployment of a new production application is a process that is strictly managed. Before an organization can transition a new application from development to production status, salesforce.com requires unit tests that validate the functionality of the application’s Force.com code routines. Submitted unit tests must cover no less than 75 percent of the application’s source code. Salesforce.com executes submitted unit tests in Force.com’s sandbox development environment to ascertain if the application code will adversely affect the performance and scalability of the multitenant population at large. The results of an individual unit test indicate basic information, such as the total number of lines executed, as well as specific information about the code that wasn’t executed by the test.
Once an application’s code is certified for production by salesforce.com, the deployment process for the application consists of a single transaction that copies all the application’s metadata into a production Force.com instance and reruns the corresponding unit tests. If any part of the process fails, Force.com simply rolls back the transaction and returns exceptions to help troubleshoot the problem.
Note: Salesforce.com reruns the unit tests for every application with each development release of Force.com to proactively learn whether new system features and enhancements break any existing applications.
After a production application is live, Force.com’s built-in performance profiler automatically analyzes and provides associated feedback to administrators. Performance analysis reports include information about slow queries, data manipulations, and sub-routines that you can review and use to tune application functionality. The system also logs and returns information about runtime exceptions to administrators to help debug their applications.
MULTITENANT SEARCH PROCESSING
Web-based application users have come to expect an interactive search capability to scan all or a selected scope of an application’s database, return up-to-date ranked results, and do it all with sub-second response times. To provide such robust search functionality for applications, Force.com uses a search engine that is separate from its transaction engine. The relationship between the two engines is depicted in the following figure. The search engine receives data from the transactional engine, with which it creates search indexes. The transactional engine forwards search requests to the search engine, which returns results that the transaction engine uses to locate rows that satisfy the search request.
DELETES, UNDELETES, AND THE RECYCLE BIN
When an app deletes a record from an object, Force.com simply marks the row for deletion by modifying the row’s IsDeleted field in MT_Data. This action effectively places the row in what is known as the Recycle Bin. Force.com lets you restore selected rows from the Recycle Bin for up to 15 days before permanently removing them from MT_Data. Force.com limits the total number of records it maintains for an organization based on the storage limits for that organization.
When an operation deletes a parent record involved in a master-detail relationship, Force.com automatically deletes all related child records, provided that doing so would not break any referential integrity rules in place. For example, when you delete a SalesOrder, Force.com automatically cascades the delete to dependent LineItems. Should you subsequently restore a parent record from the Recycle Bin, the system automatically restores all child records as well.
In contrast, when you delete a referenced parent record involved in a lookup relationship, Force.com automatically sets all dependent keys to null. If you subsequently restore the parent record, Force.com automatically restores the previously nulled lookup relationships, except for the relationships that were reassigned between the delete and restore operations.
The Recycle Bin also stores dropped fields and their data until an organization permanently deletes them or 45 days has elapsed, whichever happens first. Until that time, the entire field and all its data is available for restoration.
Historical Statistics
Years of experience have hardened Force.com into an extremely fast, scalable, and reliable cloud database service. As an illustration of Force.com’s proven capability to support Internet-scale applications, consider the following figure. Specifically notice that, over time, average page response time has decreased or held steady (a measure of performance), while average transaction volume has concurrently increased (a measure of scalability).
For more system data such as planned maintenance, historical information on transaction volume and speed, etc., visit trust.salesforce.com, the Force.com community’s home for real-time information about system performance and security.
Summary
Cloud-based services are a contemporary IT resources model that an increasing number of organizations are using to improve their time to market, reduce capital expenditures, and improve overall competitiveness in a challenging global economy. Services such as cloud application development platforms are attractive because they let businesses quickly access managed software assets on demand, and avoid the costs and complexity associated with the purchase, installation, configuration, and ongoing maintenance of an on-premises data center and dedicated hardware, software, and accompanying administrative staff.
As the foundation of salesforce.com’s enormously successful apps—such as Sales Cloud and Service Cloud, Force.com is a proven application development platform on which individual enterprises and service providers have built all types of business applications, including supply chain management, billing, accounting, compliance tracking, human resource management, and claims processing applications. Force.com’s unique, multitenant, metadata-driven architecture is engineered specifically for the cloud, and has demonstrated since 1999 that it reliably and securely supports mission-critical, Internet-scale applications. Using standards-based Web services APIs and native development tools, Force.com developers can easily build all components of a Web-based application, including the application’s data model (objects, relationships, etc.), business logic (workflows, validations, etc.), integrations with other applications, and more.
Since its inception, Force.com has been optimized by salesforce.com’s engineers for multitenancy, with features that let the system deliver unprecedented Internet scalability exceeding 1 billion complex business transactions per day. Integral system features—such as the bulk data-processing API; Apex, an external, full-text search engine; and a unique query optimizer—help make dependent applications highly efficient and scalable with little or no effort from developers.
Salesforce.com’s managed approach for the deployment of production applications ensures top-notch performance, scalability, and reliability for all applications that rely on Force.com. Salesforce.com continually monitors and gathers operational information from Force.com applications to help drive incremental improvements and new system features that immediately benefit existing and new applications.
We hope you enjoyed the series of these post about the Force.com platform, please stay tuned for other interesting articles on Salesforce.