Augmented Reality in Architecture and Design: Potentials and Challenges for Application Xiangyu Wang

international journal of architectural computing

issue 02, volume 07 309

Augmented Reality in Architecture and Design: Potentials and Challenges for Application Xiangyu Wang

Recent advances in computer interface and hardware power have fostered Augmented Reality (AR) prototypes for various architecture and design applications. More intuitive visualization platforms are necessary for efficient use of digital information nowadays in the architecture and design industries. As a promising visualization platform to address this need, this paper introduces the concept and associated enabling technologies of AR and also presents a survey of its existing applications in the area of architecture and design. Another focus of the paper is to discuss how the identified key technical issues could potentially be addressed in the context of architecture and design applications.

310

1. Introduction Mixed Reality (MR), as formally defined by Milgram et al. [1], is a special class of Virtual Reality (VR) related technologies for creating environments wherein real and virtual world objects are presented together on a single display.The terms, Augmented Reality (AR) and Augmented Virtuality (AV) are the two major subsets lying within the MR range of the RealityVirtuality (RV) continuum as shown in Figure 1. Augmented Reality, which appears in the literature usually in conjunction with the term Virtual Reality, is a technology or an environment where the additional information generated by a computer is inserted into the user’s view of real world scene [1][2]. AR can create an augmented workspace by inserting the virtual space in which users store and interact with digital contents into the physical space where people work. Such augmented workspace is realized by integrating the power and flexibility of computing environments with the comfort and familiarity of the traditional workspace [3]. By exploiting people’s visual and spatial skills, AR brings virtual information into the user’s real world view rather than pushing the user into a completely computergenerated virtual world. In the late 1990’s, several conferences specializing in this area were started, such as the International Symposium on Mixed and Augmented Reality (ISMAR) [4]. Another noteworthy avenue which is geared towards industrial settings of AR is Industrial Augmented Reality Workshop [5] which is a one-day event associated with ISMAR. More fundamentals regarding AR concept and technology could be found in [6].
Figure 1. Reality-Virtuality Continuum Adapted from [1]

Mixed Reality (MR)

Real Environment (RE)

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment (VE)

Reality-Virtuality (RV) Continuum

AR technology is envisioned to improve the current practices of architecture visualization, design process, building construction processes and engineering management systems.Architecture and design industries involve the generation of great amounts of data and information that must be accessed by numerous parties, and sometime in numerous locations. Specifically, the need for access to large amounts of design, engineering and management information in architecture and design industries creates conditions making use of AR techniques most promising by involving related personnel into the augmented workspace [7]. Human-computer interfaces that blend a view of an existing workspace with relevant design or field information should be an attractive class of technologies for architecture and design industries. Recent advances in computer interface and hardware power have fostered AR prototypes for various architecture and design applications. However, most of these lab-based prototypes were investigated by

Augmented Reality in Architecture and Design: Potentials and Challenges for Application 311

computer science/engineering societies which chose architecture and design arenas as testing areas for the purpose of proof-of-concept. Due to the lack of in-depth understanding in these practices, these efforts could hardly progress beyond the lab-phase to eventually become a usable system for real projects. Although the idea to use AR for architecture, engineering and construction dates back to the early 1996’s [8] and AR has actually matured from a pure research field into certain practical industrial applications, until now it has not been implemented as a real product in architecture and design. In contrast, architecture and design communities apparently have the knowledge of the operations/tasks that AR could potentially enhance as well as the motivation to bring in this new technology for improving the current practices. For example, Dunston and Wang [9] proposed the development of Mixed Reality-based computer interfaces, and especially Augmented Reality systems, for the architecture, engineering, and construction industries and described the technologies and principles for applying such computer interfaces to support all phases of the constructed facility project life cycle. Regardless the noted efforts, these communities have been moving very slowly toward the potential applications of this cutting-edge technology. Anders [10][11] derived and drew seven principles for the design of Mixed Reality compositions to address the basic needs served by traditional architecture from research in cognitive science, human-computer interaction design and the recognition of the multivalent, psychosomatic nature of space.These principles are grounded in human experience and provide a foundation for a polyvalent practice of architecture. The work presented in this paper holds a different perspective of discussing how Mixed and Augmented Reality has been and could be applied in architecture and design. More specifically, the technical and technological issues and challenges encountered when building and bringing AR systems into the existing practices of architectural visualization and design are focused upon.This paper consolidates existing information from an extensive bibliography of papers in the field of AR applications in architecture and design. It provides a good beginning point for guiding and encouraging the researchers who are interested in starting research in this exciting multi-disciplinary area.

2. Augmented reality in architecture and design Prior to the discussion of AR applications,Table 1, 2, and 3 presented a detailed survey for input devices, output devices, and tracking technologies, respectively. Given the potential benefits that AR could bring to the architecture and design industries, a number of lab-based research projects have been implemented. Most of noted AR applications for architecture and design embraced the ARToolkit [18] as their platform because it is simple to implement. Since ARToolkit is mostly suitable for small-scale tabletop working space, the resulting AR systems are usually regarded as Tabletop AR systems.

312

Xiangyu Wang

Input Mechanism Class 2D Input Devices 2D Imitated Controller Devices

Description

Suggestions

Examples

controlling mechanisms that use 2D input devices to generate controlling signals of six degree of freedom input mechanisms that can be programmed to map user movements into either position or rate-controlled schemes

suitable for object selecting and words input.

2D mouse and keyboard

1. more useful when implemented as position controllers and rate controllers [13] 2. well-suited for object manipulation and positioning tasks [14] 3. typically not worn, thus facilitating ease of device integration into working, desktop environments (e.g., users switch between six DOF device and keyboard, etc.) one reason spatial input devices (e.g., hand-based metaphors) for locomotion are so popular may be because no physical locomotion is required; users can travel arbitrary distances without leaving their seat, or walking at all. ten fingers can control at least ten different functionality or aspects; so it is suitable for menu manipulation.

SpaceBallTM , Magellan SpaceMouseTM, and the SpacePuckTM

1. forces the computer to translate the user's input stream, as opposed to the user translating their physical manipulations 2. physical properties of the prop or tool suggest its use and how it can be manipulated. 3. manipulating real-world objects is a familiar task and exploits existing user knowledge and skill. 4. users are immediately and continuously aware of the physical existence of the prop (as opposed to merely a visual representation which may be cluttered among other visual elements). 5. developers can provide familiar tools to user tasking, rather than an interface limited by the physical interaction of gloves, joysticks, or desktop six DOF devices. to minimize a user's cognitive load during task performance, bodycentered input devices are typically used. Another advantage of the input mechanism is that they help maintain consistency across interface tasks; walking in one part of the augmented environment should be performed no differently than walking in another part of the augmented environment. On the other hand, a poor locomotion metaphor may create a number of problems for the user.

Bat [15]

Spatial Input Devices

input mechanisms that use spatial positions and orientation as input signals and moves freely in 3D space with users, as opposed to mounted or fixed devices

Gesture Input Devices

input mechanisms that use hand gestures, which is in fact a superset of the above class: hand position and orientation provide a spatial input, and finger positions indicate gestures. input mechanisms that instrument domain-specific objects with trackers so that users may physically manipulate an object corresponding to some visually rendered virtual object

Tangible Input Devices

Body-centered Input Devices

body-centered input mechanisms that support locomotion through human’s natural means of locomotion

Flying MouseTM, BirdTM, and IsoTrakTM

DataGloveTM

[16]

 Table 1. Survey of Input Devices adapted from [12]

Augmented Reality in Architecture and Design: Potentials and Challenges for Application 313

314

Xiangyu Wang

egocentric, orthoscopic and indirect (video seethrough) view

egocentric, orthoscopic and provide a direct (optical see-through)

egocentric and orthoscopic semi-egocentric, orthoscopic and provide a direct view on the real environment

egocentric, orthoscopic and provide a direct view on the real environment egocentric, orthoscopic and provides a direct view

Monitor-Based (Handheld monitor)

Video-based ST HMD

Optical-based ST HMD

Screen-based

Head-Mounted Projectors

Spatially Augmented Reality Display

Transparent Projection Screens

Centricity

exocentric, nonorthoscopic and provide a remote view on the real environment

Display

Advantages

1. larger field of vie w and 2. higher resolution, and 1. limited and restricted viewing area 2. low resolution of the holographic film (the pattern of the holographic elements are well visible on the projection plane) 3. reduced see-through quality due to limited transparency of non-illuminated areas 1. provide a larger FOV without the application of additional lenses 2. images are always optically undistorted -even when projected onto complex non-planar surfaces 1. improved ergonomics 2. a theoretically unlimited FOV 3. a scalable resolution 4. an easier eye accommodation (because the virtual objects are typically rendered near their real world location)

1. not influenced by lighting conditions 2. relatively higher fidelity of the real environment due to direct view 3. relatively lager FOV

1. relatively higher accuracy 2. easy tracking implementation

provides a remote viewing in the real environment

Disadvantages

1. heavy and highly cumbersome 2. the integrated miniature projectors offer limited resolution and brightness 3. can only be used indoors, since they require the presence of a ceiling restrictions of the display area that is constrained to the size, shape, and color of the physical objects’ surfaces (for example, no graphics can be displayed beside the objects’ surfaces)

-----

1. direct interaction with the real environment and the graphical augmentation is not possible 2. does not support the see-through metaphor 3. small field of view that is due to relatively small monitor sizes 4. limited resolution of the merged images 1. lack in resolution 2. increased incidence of discomfort due to simulator sickness because of head-attached image plane (especially during fast head movements) 3. visual perception issues which are due to the constant image depth 4. easy user fatigue: Tethered by audio and video cabling 1. visual perception issues which are due to the constant image depth 2. easy user fatigue: Tethered by audio and video cabling, limiting user mobility to cable length and support mechanisms -----

Suggestions

-----

-----

can be integrated with optical-see through technology -----

-----

1. best suited to single user wearing a separate display, which must provide a unique perspective depending upon user location, orientation, activity, and so on 2. well-suited for complete immersion are those of architectural and interior design as well as walk-through simulation

does not support the see-through metaphor, but rather provides a remote viewing

 Table 2. Comparison of Different Types of AR Output Mechanisms Adapted from [17]

1. can be very fast 2. accurate for small volume 3. immune to magnetic, interference/distortion

currently, the best tracking technology available for large open areas is GPS

Optical

GPS

acoustic

1. no emitter required 2. not effected by external fields, materials 3. potentially small, unobtrusive sensors

Inertial

Acoustic

1. have good accuracy, responsiveness, registration, and robustness 2. can be used for force feedback so it is the best approach if a haptic interface is to be used. 3. no magnetic, line-of-sight, acoustic interference constraints 4. body-based systems are not limited to a confined workspace 1. no electro-magnetic fields 2. can be implemented fairly cheaply 3.offer good accuracy, responsiveness, robustness, and registration 4. longer ranges than magnetic trackers

1. magnetic systems have the unique advantage of being unaffected by sensor occlusion. no acoustic interference and line-of-sight constraints (particularly well-suited for hand tracking) 2. receivers are generally small and unobtrusive 3. popular, resources available and magnetic systems are relatively inexpensive and readily available

Magnetic

Mechanical

Advantages

Tracker

Disadvantages 1. distortion/interference due to metallic objects: magnetic trackers are vulnerable to distortion by metal in the environment, which exists in many desired AR application environments 2. current systems are very accurate only in small volume: as working volume increases, magnetic systems lose accuracy and registration due to impaired resistance to noise 3. accuracy dependent on distance between transmitters and receivers 1. linkages can be intrusive, heavy, and have high inertia, limiting user's movement 2. difficult to track multiple objects in same volume: mechanical linkages do not accommodate multiple users in the same working volume 3. difficult to implement for large volume: they tend to offer a limited range of operation 1. limited range 2. subject to acoustic interference such as ambient noise interference and echoes from hard surfaces, also subject to changes in air density and temperature effects 3. line-of-sight restriction 4. latency proportional to distance 1. relatively new VR technology 2. need for calibration 3. possible nonlinearity 4. inertial trackers drift with time 1. line-of-sight restriction 2. angular range is often restricted 3. interference due to other light sources 4. difficult to track multiple objects in same volume 5. optical technologies have distortion and calibration problems. 1. not suitable in indoor settings 2. relatively low accuracy

Usage Suggestions

1. line-of-sight restriction 2. angular range is often restricted 3. interference due to other light sources. 4. optical technologies show the most promise due to trends toward high-resolution digital cameras, real-time photogrammetric techniques, and structured light sources that result in more signal strength at long distances 1. GPS receivers have an accuracy of about 10 to 30 meters, which is not bad in the grand scheme of things, but isn't good enough for augmented reality, which needs accuracy measured in millimeters or smaller 2. differential GPS allows for submeter accuracy

1. subject to acoustic interference such as ambient noise interference and echoes from hard surfaces, also subject to changes in air density and temperature effects. 2. line-of-sight restriction 3. ultrasonic trackers suffer from noise and are difficult to make accurate at long ranges because of variations in the ambient temperature 1. suitable to be combined with other trackers 2. accurate orientation tracking for AR applications

1. a magnetic position tracker is well suited for use where nonferromagnetic occlusions are likely 2. magnetic position trackers have proven very effective at short range 3. magnetic systems are less suited to real time, open-room applications than either optical or acoustic systems because magnetic field strength diminishes with distance 4. magnetic trackers are vulnerable to distortion by metal in the environment, which exists in many desired AR application environments 1. no magnetic, line-of-sight, acoustic interference constraints 2. body based systems are not limited to a confined workspace 3. can be used for force feedback so it is the best approach if a haptic interface is to be used 4. they are best suited for telerobotic or constrained VR applications or in cases where user immobility is not a problem


Table 3. Tracking Technology Advantages and Disadvantages Survey Adapted from [35]

Augmented Reality in Architecture and Design: Potentials and Challenges for Application 315

 Figure 2.Technological components for an AR system

One of the applications of AR in the early phases of a design is for sketch which is a rapid and fuzzy embodiment of the architectural discussion.The sketchand+ system [19] is an experimental prototype to make a first attempt to use AR in the early architectural design stages, which could have significant impact on the quality of the entire design process.This AR prototype utilized a scribbling interface through the metaphor of a digitizer tablet and the virtual response is a 3D sketch. Sketchand+ demonstrated that collaborative tangible interaction with models could be more appropriate in early design investigation.Another example is an AR prototype implemented by Aliakseyeu et al. [20] for architectural design based on a thorough analysis of the characteristics and requirements of the early architectural design stages. The system has three basic interaction elements: the brick elements, a digitizer tablet with a digital pen, and the enhanced paper prop.The prototype system can assist in creating and editing sketches, which preserves the naturalness of the traditional way of sketching. Reviewing urban design proposals are different to that of architectural design. Generally the issues are not of detailed design but rather understanding space and spatial features. Exploring the relationship of human and the city is of major influence for urban design. BenchWorks [21], a next generation of sketchand+, was developed as an AR prototype for analyzing representational design in an urban design scale, which focused on techniques and devices necessary to create 3D models for urban design. The system was designed as a workbench, which combined optical tracking (the use of ARToolkit) with magnetic tracking. A 1.5x1.0m2 digital whiteboard was also mounted horizontally as working surface, providing a much larger interactive shared space allowing new interaction techniques. The traditional pen, paper and eraser metaphor creates the link between the traditional and virtual tools (see Figure 3).The way a user can design in BenchWorks is by representation of void and non-void space of the city, adding volumes and making notes in a tangible manner [21].

316

Xiangyu Wang


Figure 3. An augmented user is dragging a model from the toolbox into the mixed real and virtual model space [19]

Another well noted example of tabletop AR systems for urban planning is ARTHUR project [22] where optical see-through AR displays were used together with decision-support tool for architectural and urban design (see Figure 4). ARTHUR features a first-person perspective through which the rendering scale of the design model follows many conventions that designers are used to. Furthermore, the simulation of pedestrian movement inside the design allowed designers to easily recognize scale.
Figure 4. Collaboration within ARTHUR environment [22]

Dunston and Wang [9] developed an AR system called Augmented Reality Computer-Aided Drawing (AR CAD) for individual mechanical design detailing. This AR system allows users to visualize the virtual piping skid design that looks floating on a physical real tracking marker via head-mounted display (HMD). Detailed design is modeled in AutoCAD® and then sent to the customized AR program for rendering and visualizing in a real time manner (see Figure 5).The entire virtual model can be manipulated (observed from different perspectives) through physical manipulation of tracking marker. Validated benefits include enhanced spatial cognition and perception of piping designs. MIXDesign [23] provides a Mixed Reality system specifically for implementing tasks in architectural design, which developed tangible interfaces using ARToolkit patterns on a paddle and gestures. An increasingly common use of Augmented Reality is to support communication and collaboration.The AR applications for collaborative activities are called Collaborative Augmented Reality [24, 25]. Collaborative activities, especially those involving design and visualization of 3D models,

Augmented Reality in Architecture and Design: Potentials and Challenges for Application 317

 Figure 5.Views in AR CAD and AutoCAD [9]

can benefit from having multiple users simultaneously view, discuss and interact with these virtual models.The emergin emphasis on the mediation role of computers adds new technical challenges to the design of computermediated human-human interfaces. AR offers new potentials for mediating human-human interactions/communications for the entire life cycle of the engineered facility.The proper blending of the real and virtual through the attendant interaction metaphors can be geared towards enhancing group decision-making on particular life cycle activities.There are several noted efforts towards collaborative AR systems in design and planning. For instance,Wang et al. [26] developed an intuitive mixed environment called Mixed Reality-based Collaborative Virtual Environment (MRCVE) to support the collaboration and design spatial comprehension in collaborative design review sessions for mechanical contracting.The environment could be faceto-face manner or distributed over network. A significant challenge with co-located collaborative AR systems is to ensure that collaborators can establish a shared understanding of the virtual space, similar to their understanding of the physical space. In other word, it is difficult to establish common frames of reference among collaborators to describe spatially-located virtual information.The problem is that, since the virtual graphics are overlaid independently on each collaborator’s view of the world, it is difficult to ensure that each collaborator clearly understands what other collaborators point at or refer to (e.g. see the two collaborators in Figure 6). For example in MRCVE, the system attempts to tackle this problem by enabling a tracking ball. If one collaborator finds something in the model and tries to lead the discussion to the interested location in the model (e.g., “I found a clearance problem, everybody come look here”), then the tracker ball can help other collaborators to track where the identifier is trying to point at (see the two white virtual tracking balls in Figure 6). Only when the tracking ball interferences with the virtual model, collaborators recognize the accurate location of tracking marker with aid of the reference to the static virtual model. By using the tracker ball, each collaborator can track his/her hand’s position relative to the viewed virtual model by knowledge of the spatial relationship between virtual model and tracker ball. Furthermore, by overlaying a virtual ball on

318

Xiangyu Wang

the real tracking marker, collaborators are immediately and continuously aware of the physical existence of the prop (as opposed to merely a visual representation which may be cluttered among other visual elements). In the project of Studierstube [27], for example, the designers attempt to overcome this problem by rendering virtual representations of the physical pointers, which are visible to all participants. However, this does not help when the user’s gesture is with untracked hands or refers to objects descriptively.
Figure 6. MRCVE face-to-face collaboration with virtual tracking ball [26]

Except those two examples, BUILD-IT [28] is a multi-user planning tool using a 2D projection on a table top. A video camera is used to track manipulations of a small, specialized brick that can be used as a “universal interaction handler”. A second, vertical projection screen behind the table provides a 3D view of the virtual scene. Augmented Reality also provides the unique opportunity to integrate the design into the real as-built environment. For example, AR could assist in quality assurance by comparing a as-built facility with its corresponding design as well as constructability review before the actual construction. Kensek et al. [29] developed an AR system for architectural design visualization in an indoor environment by using AR for an architectural application in indoor facility management/maintenance (see Figure 7). AR systems, when used in outdoor environments, could assist in demonstrating different project stakeholders such as architects/designs/constructors/ owners what a new facility would look like at its final setting. Such augmented environment can be used to evaluate functions and esthetics of a particular design.
Figure 7.Virtual facility management data overlaid onto a real indoor setting [29]

Augmented Reality in Architecture and Design: Potentials and Challenges for Application 319

 Figure 8.Virtual bridge across a real river [30]

For outdoor AR applications, architects have been achieving similar AR effects by applying the approach of superimposing still images of proposed facilities onto a photorealistic environment. However, in terms of interactive augmentation, this approach, strictly speaking, is actually not a real AR system.The non real-time process is currently tedious and time-consuming and the results are individual and unalterable augmentations of individual photos [30]. In order to tackle this problem, Klinker et al. [30] developed an approach which could augment video sequences of large outdoor sceneries with detailed models of prestigious new architectures, such as TV towers and bridges (see Figure 8). Due to the complexity of video sequences, they were pre-recorded by employing offline and interactive calibration techniques to determine camera positions.With all calibrations set, the augmentation of the images with virtual objects was therefore performed live.This enables virtual models to be altered while being seen in the video sequence. As another well noted example,Thomas et al. [31] developed a mobile AR platform called TINMITH2 to visualize a simple extension to a building in spatial context of its outdoor final physical surroundings.

3. Issues and challenges In the design and implementation of Augmented Reality systems for industrial problems in architecture and design arenas, researchers and system developers face three major relevant challenges: extraction of industrial domain knowledge, preparation of reality model, and technological limitations.The following subsections discuss the details of each challenge and the suggested solutions.

3.1 Extraction of industrial domain knowledge As a facility is constructed, there creates large amounts of design and asbuilt information in a wide range of specialty services such as HVAC, mechanical, electrical, structural, etc. Unfortunately much information is currently represented and stored as 2D plots rather than 3D models in

320

Xiangyu Wang

practice.There apparently exists a lack of well-structured integrated 3D database which could be readily used by AR systems to support information source extraction.This is mainly due to the fact that design parties are not committed for producing the information. For example, an Augmented Reality application of “seeing into the walls” assumes that the AR system has a database storing information of the locations of all the pipes, wires and other hidden objects within the inspected facility area [2]. Unfortunately, construction contractors/subcontractors tend to maintain their own 2D database of information only relevant to their specialty services rather than create an all-inclusive large-sized 3D project model.Therefore, significant modeling (2D to 3D) and integration efforts are required to construct an AR-compatible database with appropriate formats. It is envisaged that this issue could be potentially solved through adopting Building Information Modeling (BIM) as the information source of AR systems. Furthermore, a specialized AR system geared towards one specialty service only needs to display the most relevant information to the very task in hand instead of overloading too much information onto the worker’s attention.Therefore, it is also necessary to develop an automatic and intelligent domain information extraction mechanisms (e.g., relevance, level of details) to always present the most critical and relevant information with respect to the task at hand.This will maximize the essential advantage of AR concept. Overall, modeling, integration, and extracting efforts should be taken into consideration when building an AR application.

3.2 Preparation of reality model In order to accurately register digital information into real environment, Augmented Reality systems need to obtain a precise description of the real environment: a reality model [30]. A relatively precise reality model could be defined for an accurately measured environment. One of the important issues in developing AR systems for architecture and design is to apply a systematic and accurate method to create such reality model. 3.2.1 Complexity of reality model It is known that Virtual Reality systems have high requirements for realistically rendered scenes because they completely replace the real world with a virtual counterpart. In contrast, virtual objects only supplement, not replace, the real world in Augmented Reality systems, therefore, fewer virtual objects need to be rendered [2]. Furthermore, reality models in AR systems only need to indicate geometric properties, such as easily identifiable landmarks in the scene for camera calibration, and surface shapes for occlusion handling and shadowing between real and virtual objects [30]. Accuracy actually weighs much higher than model complexity because users have an immediate quantitative appreciation of the extent of mismatches between the reality model and the input from the real scene in Augmented Reality in Architecture and Design: Potentials and Challenges for Application 321

AR systems [30]. 3D models of medium or large-sized building projects are usually very complex. Such complexity might not be a problem for off-line augmentation, however, for real-time and interactive augmentation, this could impose significant rendering problems because even powerful graphics supercomputers nowadays cannot render them at an acceptable frame rate. In this regard, model simplification mechanism is necessary to be developed to assure effective real-time rendering. 3.2.2 Methods of creating reality models There are two major methods for creating reality models.The first method is model creation from as-built information.The most straightforward approach to acquiring 3D scene is to use existing geometric models from CAD drawings [30]. Other sources of information that could be synthesized are Geographic Information Systems (GIS) and Building Information Modeling that stores data generated along the life cycle of a constructed facility (cost, schedule, quality, etc.). However, the dynamic nature of construction sites, for example, the demolishment of old buildings and infrastructure, requires approaches to generating and updating appropriate reality models for AR systems. The second method is model creation by manual measurement. Under the circumstances of no existing data, the manual approach could be adopted, which involves obtaining 3D scene information from the real world through surveying methods including Global Positioning Systems (GPS), GIS systems, laser scanning, measuring tapes, photographic images (visible or infrared), radar, ultrasound, video range camera, etc.These measured 3D image points constitute the reality model, by which digital information could be seamlessly registered into the accurate positions in the real environment. One of the things that the first approach cannot realize into reality model is to reflect and update the dynamic changes of real objects in the real environment, for example, workers, construction materials, construction equipment, etc. in the context of a construction site. Manual approach could update the reality model by keeping track of the changes of these dynamic real objects. 3.2.3 Virtual model rendering issues Rendering quality of virtual models is regarded as a critical issue since a poorly-rendered model might hinder the user’s understanding of the augmented environment. 3D architectural models created in standard modeling packages such as CAD have problems with consistent orientation of polygonal faces due to its focus on geometric modeling rather that presentation [30].The orientation information is important for calculating surface normal values for appropriate lighting purposes.There have been no effective automatic solutions to consistently integrating the faces, which in 322

Xiangyu Wang

turn requires certain level of manual work for corrections. It is also important to afford the appropriate level of realism with which digital information are rendered into the real environment. For example, if the task is to be performed under potentially dangerous conditions, where workers need to keep high situational awareness and an update of the surroundings in real time, use of solid virtual objects, large amounts of text and large size images should be avoided because they may occupy too much space in the worker’s real-world view.The wireframe format is appropriate in this case because of its see-through features but still keeping the cues for 3D shapes. In some cases, high-fidelity behavioral representations may be desirable in certain applications, which attempt to provide a high degree of realism, such as in architecture design and planning, simulation, training, etc. However, such high fidelity settings might cripple AR systems when digital information/objects consist of a very large number of polygons.

3.3 Technological limitations Technological limitations remain the major obstacle for Augmented Reality systems. For instance, AR requires highly accurate trackers because even tiny tracker errors can cause noticeable mis-registrations between real and virtual objects [2].The biggest obstacle to building effective AR systems is the requirement of accurate, long-range sensors and trackers that report the locations of the user and the surrounding objects in the environment [2].The advancement of tracking and sensing technology heavily relies on both industrial and academic efforts in the hardware domain. Besides accurate and long-range tracking, high-quality and real-time rendering [32, 33] is essential to AR systems. Achieving such rendering effects could come from two methods.Tracking based on simple sensors could be run on a wearable computing device and high-quality rendering of a complex virtual model could be achieved on a stationary graphics supercomputer [34]. Likewise, a resource-intensive tracking algorithm based on a simple reality model could run on a stationary supercomputer while simple digital information such as texts could then be rendered by wearable computing devices. Under the circumstances of little or no prior knowledge about the surrounding real environment, occlusion detection becomes a critical issue in AR systems.The issue is that the digital representations of the same object hidden into and outstanding out of a real object may present the same visually combined view to the user. Such occlusion errors or confusions might easily influence the feeling of presence the user might experience.This issue is now well investigated by the research communities in computer science and cognitive psychology.

4. Summary This paper provides a detailed discussion of Augmented Reality concept and Augmented Reality in Architecture and Design: Potentials and Challenges for Application 323

its existing and potential applications in the area of architecture and design. The key issues and challenges in the development and deployment of indoor and outdoor AR systems are also identified. Three challenges are identified for researchers and developers to face while designing AR systems for architecture and design arenas: extraction of industrial domain knowledge, preparation of reality model, and technological limitations. For industrial domain knowledge, there apparently exists a lack of well-organized integrated 3D database that could be readily used by AR systems.Therefore, it is necessary to develop an automatic and intelligent domain information extraction mechanism to always provide most critical and relevant information with respect to the task at hand. Preparation of reality model is another important issue in developing AR systems for architecture and design. Issues and systematic approaches to creating welldefined reality models are identified and discussed.The major technological limitation comes from the accuracy and robustness of available tracking systems. Current AR systems heavily rely on a strictly-controlled/defined environment which assures accurate tracking, but greatly restricts the flexibility of AR system. A robust AR system should work in all environments without the need to learn these environments in advance. Future research should focus on investigating AR systems that could work in unstructured indoor or outdoor environments.

References 1. P. Milgram and H. Colquhoun, A taxonomy of real and virtual world display integration, Mixed Reality - Merging Real and Virtual Worlds,Y. O. a. H.Tamura, Ohmsha (Tokyo) & Springer Verlag (Berlin), 1-16. (1999). 2. R.T.Azuma, A survey of Augmented Reality, Presence:Teleoperators and Virtual Environments, MIT Press, ISSN 1054-7460, 6 (4), 355-385. (1997). 3. X.Wang and P.S. Dunston, Mobile Augmented Reality for support of procedural tasks, Proceedings of Joint International Conference on Computing and Decision Making in Civil and Building Engineering, June 14-16, Montreal, Canada, 1807-1813. (2006). 4. International Workshop and Symposium on Mixed and Augmented Reality (2006). http://www.ismar06.org/, Oct. 22-25, Santa Barbara, USA 5. Industrial Augmented Reality Workshop (2006) http://www.ismar06.org/program.html#workshop, Oct. 22-25, Santa Barbara, USA 6. W. Barfield and T. Caudell, Fundamentals of Wearable Computers and Augmented Reality, Lawrence Erlbaum Associates, Mahwah, NJ.ISBN 0805829016, (2001). 7. X.Wang, P.S. Dunston, M. Skiniewski, Mixed Reality technology applications in construction equipment operator training, Proceedings of the 21st International Symposium on Automation and Robotics in Construction (ISARC 2004), September 21-25, Jeju, Korea, 393-400. (2004). 8. A.Webster, S. Feiner, B. MacIntyre,W. Massie, and T. Krueger, Augmented Reality in architectural construction, inspection, and renovation, Proceedings of ASCE Computing in Civil Engineering, Anaheim, California, June 17-19, 913-919. (1996). 9. P.S. Dunston and X.Wang, Mixed Reality-based visualization interfaces for the AEC industry, Journal of Construction Engineering and Management, American Society of Civil Engineers (ASCE), ISSN: 0733-9364, 131 (12), 1301-1309. (2005). 324

Xiangyu Wang

10. P. Anders, Designing Mixed Reality: Perception, projects, and practice, Proceedings of Conference of Association for Computer-Aided Design in Architecture (ACADIA 2007), October 1-7, Halifax, Nova Scotia, 276-283. (2007). 11. P. Anders, Cybrid Principles: Guidelines for merging physical and cyber spaces, International Journal of Architectural Computing, ISSN 1478 0771, 3 (3), 391-406. (2005). 12. J. Gabbard, A taxonomy of usability characteristics in virtual environments, MS Thesis,Virginia Polytechnic Institute and State University. (1997). 13. S. Zhai and P. Milgram, Human performance evaluation of manipulation schemes in virtual environments, Proceedings of 1993 IEEE Virtual Reality Annual International Symposium,18-22 September, Seattle,Washington. 155-161. (1993). 14. S. Zhai and P. Milgram, Human performance evaluation of isometric and elastic rate controllers in a 6 dof tracking task, Conferences & Publications on Photonics, Optics, & Imaging (SPIE), ISSN 0277-786X, 2057, 130-141. (1993). 15. C.Ware and D.R. Jessome, Using the Bat: A six-dimensional mouse for object placement, IEEE Computer Graphics and Applications, ISSN 0272-1716, 8 (6), 65-70. (1988). 16. M. Slater, M. Usoh, and A. Steed,Taking steps: the influence of a walking technique on presence in virtual reality, ACM Transactions on Computer-Human Interaction, ISSN: 1073-0516, 2 (3), 201-219. (1995). 17. O. Bimber, Interactive rendering for projection-based Augmented Reality displays, Ph.D.Thesis,Technische Universität (University of Technology) Darmstadt. (2002). 18. ARToolkit, URL: http://www.hitl.washington.edu/artoolkit/ 19. H. Seichter, Sketchand+ collaborative Augmented Reality sketching, Proceedings of the Conference on Computer-Aided Architectural Design Research in Asia (CAADRIA 2003), in A. Choutgrajank, E. Charoensilp, K. Keatruangkamala and W. Nakapan (eds), 18-20 October, Bangkok,Thailand, 209-219. (2003). 20. D. Aliakseyeu, J. Martens, and M. Rauterberg, A computer support tool for the early stages of architectural design, Interacting with Computers, ISSN: 0953-5438, 18 (4), 507-868. (2006). 21. H. Seichter, BenchWorks Augmented Reality urban design, Proceedings of the Conference on Computer-Aided Architectural Design Research in Asia (CAADRIA 2004), in H. S. Lee, J.W. Choi (eds), 28-30 April, Seoul, Korea, 937-946. (2004). 22. W. Broll, I. Lindt, J. Ohlenburg, M.Wittkämper, C.Yuan,T. Novotny, A. Fatah gen Schieck, C. Mottram, and A. Strothmann, ARTHUR: a collaborative augmented environment for architectural design and urban planning, Journal of Virtual Reality and Broadcasting, ISSN 1860-2037, 1 (1). (2004). 23. J.M.S. Dias, P. Santos, and N. Diniz,Tangible interaction for conceptual architectural design, Augmented Reality Toolkit (ART) 02, 29 September, Darmstadt Germany. (2002). 24. M. Billinghurst and H. Kato, Collaborative Mixed Reality, Proceedings of 1st International Symposium on Mixed Reality (ISMR 99), Ohmsha Ltd and Springer Verlag, March 9-11, 1999,Yokohama, Japan, 261-284. (1999). 25. A. Butz, et. al., Enveloping users and computers in a collaborative 3D Augmented Reality, Proceedings of 2nd International Workshop Augmented Reality (IWAR ’99), San Francisco, 20-21 Oct., 35-44. (1999). 26. X.Wang, D. Shin, and P.S. Dunston, Issues in Mixed Reality-based design and collaboration environments, CD Proceedings of ASCE 2003 Construction Research Congress, Honolulu, Hawaii, March 19-21, 8 pages. (2003).

Augmented Reality in Architecture and Design: Potentials and Challenges for Application 325

27. D. Schmalstieg, A. Fuhrmann, Z. Szalavri, and M. Gervautz, Studierstube: An environment for collaboration in Augmented Reality, Virtual Reality—Systems, Development and Application, ISSN: 1434-9957, 3 (1), 37-49. (1998). 28. M. Rauterberg, M. Fjeld, H. Krueger, M. Bichsel, U. Leonhardt, and M. Meier, BUILD-IT: a video-based interaction technique of a planning tool for construction and design, Proceedings of Work With Display Units (WWDU)’97, 3-5 November,Tokyo, 175-176. (1997). 29. K. Kensek, D. Noble, M. Schiler, and A.Tripathi, Augmented Reality: an application for architecture, Proceedings of 8th International Conference on Computing in Civil and Building Engineering, Stanford, CA, August 14-16, 294-301. (2000). 30. G. Klinker, D. Stricker, and D. Reiners, Augmented Reality for exterior construction applications, Fundamentals of Wearable Computers and Augmented Reality, ISBN: 978-0-8058-2902-0, 379-427. (2001). 31. B.Thomas,W. Piekarski, B. Gunther, Using Augmented Reality to visualize architecture designs in an outdoor environment, In DCNET1999 - Design Computing on the Net, November 30-December 3, Sydney, NSW, Australia. (1999). 32. D. Reiners, High-quality realtime rendering for virtual environments, Diplomarbeit,TU Darmstadt. (1994). 33. J. Rohlf and J. Helman. IRIS performer:A high performance multiprocessing toolkit for real-time 3D Graphics, Proceedings of ACM Siggraph’94, Orlando, FL, 381-395. (1994). 34. T. Starner, S. Mann, B. Rhodes, J. Levine, J. Healey, D. Kirsch, R.W. Picard and A. Pentland, Augmented Reality through wearable computing, Presence, Special Issue on Augmented Reality, ISSN 1054-7460 6 (4), 386-398. (1997). 35. R. Holloway and A. Lastra,Virtual Environments: A survey of the technology, SIGGRAPH’95 Course, 8 (A), 1-40. (1995).

Xiangyu Wang Lecturer, Design Lab, Faculty of Architecture, Design and Planning, The University of Sydney, Sydney, Australia; International Scholar, Department of Housing and Interior Design, Kyung Hee University, Seoul, Korea. [email protected]

326

Xiangyu Wang