Meta-computing and Grid

This term appeared with the development of high-speed network infrastructure in the early 90's and belonged to pooling of several heterogeneous computing resources in network organization for solving one assignment. The main goal of developing a meta-computer was the optimal distribution of work on computing systems with different architectures and different capacity. 
For example, preliminary data processing and grid generation for the calculation could be performed on the workstation assigned for user, the basic simulation on the conveyor-vector supercomputer, solving large systems of linear equations - on-masive-paralllel system, and results visualization - on a special graphic station. 


It is obvious that the best choice is to work with sorting and searching assignments on meta-computers, where computational nodes are not interacting with each other and the main part of work is performed in autonomic mode. (Offline)
This is a basic scheme in this case- a special agent, located on a computational node (users computer), determines computer’s delay, connects with the meta-computer’s node, which administrates and gets another part of work from it (a new sorting area). After the end of calculation of this part, the computational node sends back a report about executed enumeration or a signal that a goal of search is achieved.

Parallel computing in 1980`s focused efforts of its researchers on developing of algorithms, programs and architectures that support simultaneous operation of several processors. During 1980 and 1990`s the software development for parallel computers concentrates on providing powerful mechanisms for managing connections between processors and on developing and creating environments for parallel machines. Parallel Virtual Machine (PVM), Message Passing Interface(MPI), High Performance Fortran (HPF), and OpenMP were developed for connection support in scalable applications. Also a number of successful paradigms to enable great potential of delimited and distributed memory architecture was developed

What is a Grid-computing?

Further researches in the area of meta-computing technologies were developed in direction of providing numerous users with simultaneous access to a large number of computational resources (up to several thousands computers in local or global networks), of data storage, scientific equipment, computer networks, etc. In the same time their quantity turned into the quality and created a new term- Grid-Computing.

Originally Grid was considered to be more efficient in parallel computing development- from tight linked groups to geographically distributed systems. However in practice Grid was used more like a platform for integration of free-connected applications (some of their units could work on parallel machine) and for connection of distributed resources (storage, calculation, visualization, instruments). The I-WAY (Information Wide-Area Year) is considered to be the first modern Grid, which was developed as experimental demonstrational project in 1995 and which pooled 17 computational nodes, connected via high-speed backbone based on ATM technology (1.5 Mb/s- 9.6 Mb/s) into the national distributed network. A distributed file system and recourse broker, which had a central node, that administrated the network and agents were implemented on other nodes.
The development of infrastructure and applications for I-WAY provided meaningful and powerful experience for the first-generation of Grid systems.
These are some of the achievements:
• Transformation of the computer into a device, designed for many users and creation timesharing systems
• interaction organization on processes level by
-total address space management 
- Distributed shared memory (DSM) 
- Message passing (MPI, PVM) for heterogeneous systems support and 
work in a local /global network  
•access model conversion to software and hardware computer resources on distributed computing environment- Grid user can work with its resources as well as with resources of its own computer.
• Development of the following scheme: 
- The agent on the machine determines the IDLE and connects to the server that manages computations; 
- Server that manages sends a new task (data for processing, an area in the search space) to the agent; 
- After computing completion the results are sent back to the server; 
• support for dynamic environment configuration, when Grid System 
has no permanent structure and is being dynamically organized with geographically distributed resources, temporarily delegated by their actual owners 
(Virtualization of meta-computer) 
• Organization of uniform access to computing resources of numerous computers in local or global network
• problem solving that allow decomposition to large number of small, independent subtasks.
Further development and generalization of meta-computing ideas to a wider range of computing resources and tasks / applications has required an immediate solution of a wide range of issues related to data transfer, safety, task management, data access, resource search, access to them and other. Approaches to the development of such Grid-system were described in [Foster I. The Grid: Blueprint for a New Computing Infrastructure / Ian Foster and Carl Kesselman. - Morgan Kaufmann Publishers.-San Francisco.-1998 .- 286 p..], And it can be considered as a representative of second generation Grid systems, which have three main features:

Heterogeneity: Grid uses a variety of resources that are originally heterogeneous, and could encompass multiple administrative domains through a potentially global space; 
Scalability: Grid could develop from few resources to thousands. This raises the potential problem of reduced productivity with the growth and spreading of Grid. Therefore, applications should be developed to use available resources, and significantly improved features of users’ authentication; 
Adaptability: resource failure in Grid is most likely a rule not an exception. 
In fact, with too many resources in the Grid, probability of resource failure is rather high. Resource managers or applications should dynamically adapt so that they can get maximum efficiency from available resources and services. 

The middleware PHZ is used to hide the heterogeneous nature of Grid and to create a homogeneous environment for users and applications, providing a number of standardized interfaces and numerous services. The PHZ lies between the operating system and applications, providing applications with a lot of services that are necessary for correct operation in distributed heterogeneous environments.

Developing of the PHZ was meant to be secession from meta-computing and creating new wave in Grid technologies. Grid infrastructure can consist of any type of network resources, starting with computers (computational resources) to data storage and special scientific instruments, Grid application usually require enormous data volumes and calculations. The need in creation of new type of Grid system software originated, system where resources are managed and users’ accounting is being done centralized in a single trusted domain, provides:
- great number of trust domains that use different security mechanisms
- A dynamic set of resources managed autonomously within one 
Domain
- A dynamic set of users who have accounts in a number of
domains 
- Access to multiple resources in the same time during the process of calculations from different 
Domains 
- processes launching on multiple resources 
- organization of interaction between resources from different domains during computing
- Interaction between running processes. 

We should refer to achievements of second generation Grid following:
• Globus project origin, which is directed at formalization and developing of toolkit for basic services for
- Resource Management 
- calculation management 
- Data Management 
- Security; 
• Develop and implement overall Grid infrastructure in the form of basic independent services 
• From resources and applications (authentication, authorization, search and resource allocation, event notification, resource accounting, remote data access, failures detection, etc.) 
• Demonstration of opportunities for integrating basic services into more complex high-level services 
• developing a range of software components and utilities that perform tasks for remote execution resources (supercomputers, clusters) through the already existing resource management systems(usually a batch processing), task control and task management; collection and access to various information about system and its components, etc. , with their help high-level services can be provided for users, applications, resource planners, brokers. 
• Initiation of several projects such as Cactus, WebFlow, DataGr³d on creation of large-scale computing and information Grid recourses networks for data analysis.

However the second generation of the PHZ did not provide full interoperability of the created Grid software that had been an important condition for realization of large-scale computing. To develop new Grid-applications, it is better to repeatedly use all existing components and information resources, and to operate them flexibly.
The third generation Grid systems have two key features- emphasis on the adoption of service-oriented model and increasing attention to metadata. In fact, the service-oriented approach suggests that the flexible Grid resources’ usage in Grid applications would require information about functional capabilities, specifications and interfaces of various components, and this information must be coherent for computer processing.
New terms - "distributed collaboration" and "virtual organization" were introduced. In the third generation more integral picture of Grid computing realization was formed and one can say that it is more about the infrastructure for e-science than about improving of the existing technology. Grid can provide support for remote communication for scientists in real time.
The infrastructure for support of distributed resources is especially important- it has many key services: security, scalability and management, registration and search, and message-oriented Web-services interfaces that provide powerful mechanisms for cooperation. All major Grid-services and infrastructure guarantee cooperation and are extremely important for the society.
The third Grid-system generation aims to create the "general research area» («collaboratory»), where local researchers could perform studies without regard to geographic location through interaction with colleagues using shared toolset, data and computing resources and addressing for the information to digital libraries. The Grid2003 project can be a successful example of these efforts, it is developed to combine powerful computing centers in the U.S.; the EGEE project (Enabling Grids for E-science in Europe), aims to combine national, regional and thematic Grid-systems into a coherent Grid-infrastructure to support scientific research; the Access Grid project, covers resource collection for support peoples’ cooperation through Grid, including large-scale distributed meetings and training.
Grid system of the third generation moves forward to Semantic Grid systems, based on the usage of metadata and ontology, where the information is considered not only as data that is important, but as knowledge that are being extracted, used, presented, published and maintained, to assist e-scientists in achieving their specific goals. Knowledge is considered as information that is applied to achieve the goal, solve the assignment or decision. 
The Semantic Grid covers all three conceptual layers of Grid: knowledge, information and computing / data. These additional layers will eventually provide abundant, seamless and widespread access that will apply to the globally distributed heterogeneous resources. One of the standard techniques used in the management of knowledge is a knowledge portal development. According to The standard definition, portal – is a working Web-based application that provides tools for storage, adaptation and personalization of information. 
The most significant achievements of the third generation Grid systems that are obtained today:
• •Merging of Grid-technology and Web-services technologies, the generation of Grid service as a special expansion of Web-service, by the support of Grid-service copy, with the condition and with possible limited lifetime; data services are related with each copy of Grid-service, it means that information is structured as a set of named XML-elements typified (service data elements, SDE 
• providing interoperability of different implementations of services, determination of standardized service interfaces (OGSI), protocol definition (CQ) to get a particular interface, an agreement on a standard set of supported protocols, where for each standard interface a set of service data elements is defined, that have to be supported by any service copy instance that implements this interface;
• the cooperation of GT, IBM and other companies offered a set of specifications under the name WS-Resource Framework (WSRF), which is based on the same architecture -OGSA, on generally accepted standards for Web-services, and a lot of elements OGSI are present, but it uses other terminology and expands OGSI opportunities; 
• creation of the Globus Toolkit GT4 WSRF (Grid software) –as an open implementation of WSRF and of the development tools, client and server applications in Java, C + + and Python; while GT4 is not fully compatible with the previous GT3. 

History of Globus Toolkit demonstrates the evolution of leading-edge technology from a purely academic project with a narrow range of users to a generally accepted standard, which is supported in the IT-industry around the world.
Even though the level of today's Grid is quite high and most software is available and is used directly, it still has deficiency in many significant aspects that will ensure effective access, expansion and usage of system resources. These are the problems that require further elaboration: 
Information services - we need mechanisms that are used to store information about resources, because the Grid needs to provide remote, rapid, reliable, secure and scalable services.
• Information about resources - for the correct performance, Grid will need all kinds of information, from the name of the resource and data security requirements to the applications and user settings. It is important that this information was understandable, easily interpreted and could be used by every service that would need it.
• Resource Identification – we need to have mechanisms to locate resources within globally distributed system, if there is a unique resource name or its characteristics. Services are resources. Some resources can be stored, some may be transient, and some can be created on demand.
• • Synchronization and coordination – there should be mechanisms for managing a complex sequence of calculations on a great number of resources. The process description may become necessary and event-Oriented Infrastructure availability as well, which involves planning at various levels, including metaplanning and information streams.
• Resistance to system failure - the environment must cope with failure of software components and equipment, as well as with access problems in general, and exception’s handling should be considered in this multi-user multi-system
• • Security - authentication, authorization, warranty, and accounting mechanisms should be set in place, and they must operate within the context of automation and scale, that increases.
• Parallelism and consistency – there is a need to support the appropriate level of consistency of data in parallel, heterogeneous environment. For some applications a less well-defined sequence can be sufficient.
• Capacity –a demand to be able to cope with global access to resources, through caching and replication. 
It can look attractive to move code (or service) to the data (perhaps with scripts or mobile agents) but it includes a number of problems.
• Diversity - the need to work with a lot of hardware, with different software and information resources, and to do it through multiple organizations with different administrative structures.
• Scalability - the system should be able to expand the number and size of services and applications without requiring human intervention. Grid needs automation and ideal self-organization for it.
• Interfaces-the requirement to use Grid, despite specific middleware in various infrastructure and platforms, that will allow working in mixed Grid environment;
Distributed computing technology that existed before the emergence of Grid-technologies do not fulfill these conditions. For example, Internet technology is intended for communication and information exchange between computers, but do not provide an integrated approach for the coordinated usage of resources on multiple computing systems. Meta-computing provides parallel solution for a specific assignment on a number of computational resources.
Business Technologies focus only on the problems of joint information usage, more often via centralized servers. Distributed computing technology with the amount of enterprise, such as CORBA or Enterprise Java, allows the resource sharing only within one organization. Open Group's Distributed Computing Environment (DCE) facilities support secure resource sharing of multiple sites, but the majority of virtual organizations find these tools onerous and inflexible. 
Storage service provider (storage service provider - SSP) and application service provider (application service provider - ASP) allow to use data storage and computing by a third party, with some restrictions (eg, SSP resources are available to users only through virtual private networks (virtual private network - VPN). Thus, the current technologies do not allow combining different types of resources, or do not provide flexibility in management of shared connections. 

It is important to emphasize, while talking about usage of Grid-technologies that they allow not only to solve scientific and practical assignments that were difficult because of the solving duration. Grid provides a basis for a new organization of science, for high tech industries, for social life, and allows to manage the resources of society more effectively and securely. Grid provides a platform, with its help users get access to the combined computing, data storage and network resources – so they can to execute their applications for distributed data processing. This provides us with a functional environment that allows users to analyze data, share results with colleagues and use information about the data without established limits and geographic boundaries.
We think that, current grid-computing and meta-computing from the 90’s can not be considered identical. Meta-computing is related to the parallelization of some problems on the set of processors (classical cluster is a typical example) and Grid-computing –is related to the best fitting and organization , to the implementation of various assignments in the virtual field of different resources, ,where computing resources can be monoprocessor. Grid-technologies and parallel-distributed computing technologies do not match.
If one can not see the difference between the Grid-computing and meta-computing it is the same as to call modern “shuttle” in old way- “airplane” next to “kukuruznik”, its like having troubles recognizing aviation and space technologies and yet to talk about the development of “aeronautics” in space.
Or even worse it is like calling "kukuruznyk" a "shuttle". This can lead to confusion in communication with foreign experts (that have ideas about development and correlation of Grid-computing and meta-computing, close to the stated above) who can not understand how to build a Grid-system (not a meta- computer!) on the department without using the middleware PHZ. 



Ukrainian (UA)Russian (CIS)English (United Kingdom)